StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POLAMP: How to create .Zip of large files for the user on the fly, without disk/CPU thrashing
text
Body
copied!<p>Often a web service needs to zip up several large files for download by the client. The most obvious way to do this is to create a temporary zip file, then either <code>echo</code> it to the user or save it to disk and redirect (deleting it some time in the future).</p> <p>However, doing things that way has drawbacks:</p> <ul> <li>a initial phase of intensive CPU and disk thrashing, resulting in...</li> <li>a considerable initial delay to the user while the archive is prepared</li> <li>very high memory footprint per request</li> <li>use of substantial temporary disk space</li> <li>if the user cancels the download half way through, all resources used in the initial phase (CPU, memory, disk) will have been wasted</li> </ul> <p>Solutions like <a href="http://pablotron.org/software/zipstream-php/" rel="noreferrer">ZipStream-PHP</a> improve on this by shovelling the data into Apache file by file. However, the result is still high memory usage (files are loaded entirely into memory), and large, thrashy spikes in disk and CPU usage.</p> <p>In contrast, consider the following bash snippet:</p> <pre><code>ls -1 | zip -@ - | cat > file.zip # Note -@ is not supported on MacOS </code></pre> <p>Here, <code>zip</code> operates in streaming mode, resulting in a low memory footprint. A pipe has an integral buffer – when the buffer is full, the OS suspends the writing program (program on the left of the pipe). This here ensures that <code>zip</code> works only as fast as its output can be written by <code>cat</code>.</p> <p>The optimal way, then, would be to do the same: replace <code>cat</code> with a web server process, <em>streaming</em> the zip file to the user with it created on the fly. This would create little overhead compared to just streaming the files, and would have an unproblematic, non-spiky resource profile.</p> <p>How can you achieve this on a LAMP stack?</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload