Jul 20, 2023

Splits stdin of unknown size onto chunks packed as a TAR to stdout

Splits stdin of unknown size onto chunks of given size and collects such chunk files as a TAR archive what is written to stdout.

Initial motivation was to use it as a packer of a data stream what pipes it to https//tarsnap.com for backup purposes, with a requirement of low and more predictable memory consumption. And the problem solved here is that TAR file header should specify size of a file, but it’s unknown due to the nature of input data stream.

The following real case could be an example $ pg_dump … | tar_stream_chunker … | tarsnap -c … @-

Checkout these related ports:
  • Zutils - Utilities for searching in bzip2, gzip, lzip, and xz archives
  • Zstr - C++ header-only ZLib wrapper
  • Zstd - Fast real-time compression algorithm
  • Zpaqfranz - Swiss army knife for the serious backup manager
  • Zopfli - Zopfli Compression Algorithm
  • Zoo - Manipulate archives of files in compressed form
  • Zlib-ng - Fork of the zlib data compression library
  • Zipper - Tool for inspecting the contents of a compressed archive
  • Zipmix - Produce .ZIP file from two other ones with the best compressed files
  • Zip - Create/update ZIP files compatible with PKZIP
  • Zchunk - Compressed file format that splits the file into independent chunks
  • Xpk - The eXternal PacKer (XPK) library system
  • Xmill - Efficient compressor for XML
  • Xdms - Tool for decompressing Amiga DMS files
  • Xarchiver - Desktop-agnostic GTK frontend to various archiving tools