Databases
StackOverflow2013
Postsdbo
POPython: how to stream/pipe data out of gzip compression?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POPython: how to stream/pipe data out of gzip compression?
    primarykey
    data
    text
    <p>I need to do something like this, but in python:</p> <pre><code>dd if=/dev/sdb | gzip -c | curl ftp upload </code></pre> <p>I can't use the entire command with Popen because:</p> <ol> <li>I need non-blocking operation</li> <li>I need progress information (tried looping through proc.stderr to no avail)</li> </ol> <p>The other big thing is I can't create a compressed gzip file in memory or on disk prior to uploading.</p> <p>So this is about what I'm looking to figure out how to do, with gzip_stream_of_strings(input) being the unknown:</p> <pre><code>import os, pycurl filename = '/path/to/super/large/file.img' filesize = os.path.getsize(filename) def progress(dl_left, dl_completed, ul_left, ul_completed): return (ul_completed/filesize)*100 def main(): c = pycurl.Curl() c.setopt(c.URL, 'ftp://IP/save_as.img.gz') c.setopt(pycurl.NOPROGRESS, 0) c.setopt(pycurl.PROGRESSFUNCTION, progress) c.setopt(pycurl.UPLOAD, 1) c.setopt(pycurl.INFILESIZE, filesize) c.setopt(pycurl.USERPWD, 'user:passwd') with open(filename) as input: c.setopt(pycurl.READFUNCTION, gzip_stream_of_stings(input)) c.perform() c.close() </code></pre> <p>Any help is greatly appreciated!</p> <p><strong>EDIT: Here's the solution:</strong></p> <pre><code>from gzip import GzipFile from StringIO import StringIO CHUNCK_SIZE = 1024 class GZipPipe(StringIO): """This class implements a compression pipe suitable for asynchronous process. Credit to cdvddt @ http://snippets.dzone.com/posts/show/5644 @param source: this is the input file to compress @param name: this is stored as the name in the gzip header @function read: call this to read(size) chunks from the gzip stream """ def __init__(self, source = None, name = "data"): StringIO.__init__(self) self.source = source self.source_eof = False self.buffer = "" self.zipfile = GzipFile(name, 'wb', 9, self) def write(self, data): self.buffer += data def read(self, size = -1): while ((len(self.buffer) &lt; size) or (size == -1)) and not self.source_eof: if self.source == None: break chunk = self.source.read(CHUNCK_SIZE) self.zipfile.write(chunk) if (len(chunk) &lt; CHUNCK_SIZE) : self.source_eof = True self.zipfile.flush() self.zipfile.close() break if size == 0: result = "" if size &gt;= 1: result = self.buffer[0:size] self.buffer = self.buffer[size:] else: result = self.buffer self.buffer = "" return result </code></pre> <p>Used like so:</p> <pre><code>with open(filename) as input: c.setopt(pycurl.READFUNCTION, GZipPipe(input).read) </code></pre>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USBrian
    1. USBrian
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POPython: how to stream/pipe data out of gzip compression?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POPython: how to stream/pipe data out of gzip compression?
      1. This table or related slice is empty.
      1. VTUpMod
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload