StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POHow does the ability to compress a stream affect a compression algorithm?
text
Body
copied!<p>I recently backed up my soon-to-expire university home directory by sending it as a tar stream and compressing it on my end: <code>ssh user@host "tar cf - my_dir/" | bzip2 > uni_backup.tar.bz2</code>.</p> <p>This got me thinking: I only know the basics of how compression works, but I would imagine that this ability to compress a stream of data would lead to poorer compression since the algorithm needs to finish handling a block of data at one point, write this to the output stream and continue to the next block.</p> <p>Is this the case? Or do these programs simply read a lot of data into memory compress this, write it, and then do this over again? Or are there any clever tricks used in these “stream compressors”? I see that both <em>bzip2</em> and <em>xz</em>'s man pages talk about memory usage, and <em>man bzip2</em> also hints to the fact that little is lost on chopping the data to be compressed into blocks:</p> <blockquote> <p>Larger block sizes give rapidly diminishing marginal returns. Most of the compression comes from the first two or three hundred k of block size, a fact worth bearing in mind when using bzip2 on small machines. It is also important to appreciate that the decompression memory requirement is set at compression time by the choice of block size.</p> </blockquote> <p>I would still love to hear if other tricks are used, or about where I can read more about this.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload