P5-net-cascadecopy

Jul 20, 2023

Rapidly propagate files to many servers in multiple locations

This module implements a scalable method of quickly propagating files to a large number of servers in one or more locations via rsync or scp.

This module and the included script, ccp, take a much more efficient approach that is Olog n. Once the files are been copied to a remote server, that server will be promoted to be used as source server for copying to remaining servers. Thus, the rate of transfer increases exponentially rather than linearly.

Servers can be specified in groups e.g. datacenter to prevent copying across groups. This maximizes the number of transfers done over a local high-speed connection LAN while minimizing the number of transfers over the WAN.

The number of multiple simultaneous transfers per source point is configurable. The total number of simultaneously forked processes is limited via ProcQueue, and is currently hard coded to 32.



Checkout these related ports:
  • Zyre - Framework for proximity-based peer-to-peer applications
  • Zsync - File transfer program
  • Zmap - Internet scanner
  • Zillion - Distributed computing project
  • Zerotier - Network virtualization everywhere
  • Zebra-server - Z39.50/SR server software
  • Yptransitd - Replacement for nss_ldap
  • Yggdrasil - Experimental end-to-end encrypted self-arranging IPv6 network
  • Yconalyzer - TCP Traffic Analyzer
  • Yazproxy - Powerful general purpose Z39.50/SRW/SRU proxy
  • Yaz - Z39.50/SR client and API library
  • Yaz++ - C++ toolkit for development of Z39.50v3 clients and servers
  • Yate - Yet Another Telephony Engine
  • Yaph - Yet Another Proxy Hunter (proxy scanner)
  • Yami4 - Messaging library for distributed systems