Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>You can use <code>popen()</code> <a href="http://www.php.net/manual/en/function.popen.php" rel="nofollow noreferrer">(docs)</a> or <code>proc_open()</code> <a href="http://www.php.net/manual/en/function.proc-open.php" rel="nofollow noreferrer">(docs)</a> to execute a unix command (eg. zip or gzip), and get back stdout as a php stream. <code>flush()</code> <a href="http://www.php.net/manual/en/function.flush.php" rel="nofollow noreferrer">(docs)</a> will do its very best to push the contents of php's output buffer to the browser.</p> <p>Combining all of this will give you what you want (provided that nothing else gets in the way -- see esp. the caveats on the docs page for <code>flush()</code>).</p> <p>(<strong>Note</strong>: don't use <code>flush()</code>. See the update below for details.)</p> <p>Something like the following can do the trick:</p> <pre><code>&lt;?php // make sure to send all headers first // Content-Type is the most important one (probably) // header('Content-Type: application/x-gzip'); // use popen to execute a unix command pipeline // and grab the stdout as a php stream // (you can use proc_open instead if you need to // control the input of the pipeline too) // $fp = popen('tar cf - file1 file2 file3 | gzip -c', 'r'); // pick a bufsize that makes you happy (64k may be a bit too big). $bufsize = 65535; $buff = ''; while( !feof($fp) ) { $buff = fread($fp, $bufsize); echo $buff; } pclose($fp); </code></pre> <hr> <p>You asked about "other technologies": to which I'll say, "anything that supports non-blocking i/o for the entire lifecycle of the request". You could build such a component as a stand-alone server in Java or C/C++ (or any of many other available languages), <em>if</em> you were willing to get into the "down and dirty" of non-blocking file access and whatnot.</p> <p>If you want a non-blocking implementation, but you would rather avoid the "down and dirty", the easiest path (IMHO) would be to use <a href="http://nodejs.org/" rel="nofollow noreferrer">nodeJS</a>. There is plenty of support for all the features you need in the existing release of nodejs: use the <code>http</code> module (of course) for the http server; and use <code>child_process</code> module to spawn the tar/zip/whatever pipeline.</p> <p>Finally, if (and only if) you're running a multi-processor (or multi-core) server, and you want the most from nodejs, you can use <a href="https://github.com/davglass/spark2" rel="nofollow noreferrer">Spark2</a> to run multiple instances on the same port. Don't run more than one nodejs instance per-processor-core.</p> <hr> <p><strong>Update</strong> (from Benji's excellent feedback in the comments section on this answer)</p> <p><strong>1.</strong> The docs for <code>fread()</code> indicate that the function will read only up to 8192 bytes of data at a time from anything that is not a regular file. Therefore, 8192 may be a good choice of buffer size. </p> <p>[editorial note] 8192 is almost certainly a platform dependent value -- on most platforms, <code>fread()</code> will read data until the operating system's internal buffer is empty, at which point it will return, allowing the os to fill the buffer again asynchronously. 8192 is the size of the default buffer on many popular operating systems. </p> <p>There are other circumstances that can cause fread to return even less than 8192 bytes -- for example, the "remote" client (or process) is slow to fill the buffer - in most cases, <code>fread()</code> will return the contents of the input buffer as-is without waiting for it to get full. This could mean anywhere from 0..os_buffer_size bytes get returned.</p> <p>The moral is: the value you pass to <code>fread()</code> as <code>buffsize</code> should be considered a "maximum" size -- never assume that you've received the number of bytes you asked for (or any other number for that matter). </p> <p><strong>2.</strong> According to comments on fread docs, a few caveats: <a href="http://php.net/manual/en/security.magicquotes.php" rel="nofollow noreferrer">magic quotes</a> may interfere and must be <a href="http://php.net/manual/en/security.magicquotes.disabling.php" rel="nofollow noreferrer">turned off</a>.</p> <p><strong>3.</strong> Setting <code>mb_http_output('pass')</code> <a href="http://php.net/manual/en/function.mb-http-output.php" rel="nofollow noreferrer">(docs)</a> may be a good idea. Though <code>'pass'</code> is already the default setting, you may need to specify it explicitly if your code or config has previously changed it to something else.</p> <p><strong>4.</strong> If you're creating a zip (as opposed to gzip), you'd want to use the content type header:</p> <pre><code>Content-type: application/zip </code></pre> <p>or... 'application/octet-stream' can be used instead. (it's a generic content type used for binary downloads of all different kinds):</p> <pre><code>Content-type: application/octet-stream </code></pre> <p>and if you want the user to be prompted to download and save the file to disk (rather than potentially having the browser try to display the file as text), then you'll need the content-disposition header. (where filename indicates the name that should be suggested in the save dialog):</p> <pre><code>Content-disposition: attachment; filename="file.zip" </code></pre> <p>One should also send the Content-length header, but this is hard with this technique as you don’t know the zip’s exact size in advance. <strong>Is there a header that can be set to indicate that the content is "streaming" or is of unknown length? Does anybody know?</strong></p> <hr> <p>Finally, here's a revised example that uses all of @<a href="https://stackoverflow.com/users/196312/benji-xvi">Benji's</a> suggestions (and that creates a ZIP file instead of a TAR.GZIP file):</p> <pre><code>&lt;?php // make sure to send all headers first // Content-Type is the most important one (probably) // header('Content-Type: application/octet-stream'); header('Content-disposition: attachment; filename="file.zip"'); // use popen to execute a unix command pipeline // and grab the stdout as a php stream // (you can use proc_open instead if you need to // control the input of the pipeline too) // $fp = popen('zip -r - file1 file2 file3', 'r'); // pick a bufsize that makes you happy (8192 has been suggested). $bufsize = 8192; $buff = ''; while( !feof($fp) ) { $buff = fread($fp, $bufsize); echo $buff; } pclose($fp); </code></pre> <hr> <p><strong>Update</strong>: (2012-11-23) I have discovered that calling <code>flush()</code> within the read/echo loop can cause problems when working with very large files and/or very slow networks. At least, this is true when running PHP as cgi/fastcgi behind Apache, and it seems likely that the same problem would occur when running in other configurations too. The problem appears to result when PHP flushes output to Apache faster than Apache can actually send it over the socket. For very large files (or slow connections), this eventually causes in an overrun of Apache's internal output buffer. This causes Apache to kill the PHP process, which of course causes the download to hang, or complete prematurely, with only a partial transfer having taken place.</p> <p>The solution is <em>not</em> to call <code>flush()</code> at all. I have updated the code examples above to reflect this, and I placed a note in the text at the top of the answer.</p>
    singulars
    1. This table or related slice is empty.
    1. POLAMP: How to create .Zip of large files for the user on the fly, without disk/CPU thrashing
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USCommunity
    1. USLee
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POLAMP: How to create .Zip of large files for the user on the fly, without disk/CPU thrashing
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COThanks for this, I have tested it fairly extensively, and there is little overhead – this appears to be a Good Solution.
      singulars
      1. PO
      1. USBenji XVI
    2. COA couple of minor issues: (1.) Per docs (and bug reports that describe how the documentation is wrong!), `fread` will read only up to 8192 bytes of data at a time from anything that is not a regular file. 8192 may therefore be a good choice of buffer size. (2.) According to comments on `fread` docs, a few caveats: magic quotes may interfere and must be turned off; setting mb_http_encoding('pass')` may be a good idea. (3.)Perhaps as this question is specifically about zip, (which is the only option for serving users cross-platform), change those parts of the code?
      singulars
      1. PO
      1. USBenji XVI
    3. COUseful headers: `"Content-type: application/zip"` (or `application/octet-stream`), and `Content-disposition: attachment; filename="file.zip"`. One should also set Content-length, but this is hard with this technique as you don’t know the zip’s exact size in advance.
      singulars
      1. PO
      1. USBenji XVI
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload