Databases
StackOverflow2013
Postsdbo
POPython HTTPConnection file send with httplib, retrieving progress
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POPython HTTPConnection file send with httplib, retrieving progress
    primarykey
    data
    text
    <p>In a django app I'm using a third-party Python <a href="http://wiki.blip.tv/index.php/REST_Upload_Python_Script" rel="nofollow noreferrer">script</a> to allow users to upload files to blip.tv through httplib.HTTPConnection.send on an EC2 instance. Since these files are generally large I will use a message queue to process the upload asynchronously (RabbitMQ/Celery), and give feedback on the progress to the user in the frontend. </p> <p>The httpconnection and send are done in this part of the script:</p> <pre><code>host, selector = urlparts = urlparse.urlsplit(url)[1:3] h = httplib.HTTPConnection(host) h.putrequest("POST", selector) h.putheader("content-type", content_type) h.putheader("content-length", len(data)) h.endheaders() h.send(data) response = h.getresponse() return response.status, response.reason, response.read() </code></pre> <p>The getresponse() is returned after the file transfer is completed, how do I write out the progress (assuming something with stdout.write) so I can write this value to the cache framework for display (djangosnippets 678/679)? Alternatively if there is a better practice for this, I'm all ears! </p> <p><strong>EDIT:</strong></p> <p>Since I've gone with urllib2 and used a tip from <a href="https://stackoverflow.com/questions/5925028/urllib2-post-progress-monitoring">this question</a> to override the read() of the file to get the upload progress. Furthermore I'm using poster to generate the multipart urlencode. Here's the latest code:</p> <pre><code>from poster.encode import multipart_encode from poster.streaminghttp import register_openers def Upload(video_id, username, password, title, description, filename): class Progress(object): def __init__(self): self._seen = 0.0 def update(self, total, size, name): self._seen += size pct = (self._seen / total) * 100.0 print '%s progress: %.2f' % (name, pct) class file_with_callback(file): def __init__(self, path, mode, callback, *args): file.__init__(self, path, mode) self.seek(0, os.SEEK_END) self._total = self.tell() self.seek(0) self._callback = callback self._args = args def __len__(self): return self._total def read(self, size): data = file.read(self, size) self._callback(self._total, len(data), *self._args) return data progress = Progress() stream = file_with_callback(filename, 'rb', progress.update, filename) datagen, headers = multipart_encode({ "post": "1", "skin": "xmlhttprequest", "userlogin": "%s" % username, "password": "%s" % password, "item_type": "file", "title": "%s" % title.encode("utf-8"), "description": "%s" % description.encode("utf-8"), "file": stream }) opener = register_openers() req = urllib2.Request(UPLOAD_URL, datagen, headers) response = urllib2.urlopen(req) return response.read() </code></pre> <p>This works though only for file path inputs, rather than an InMemoryUploadedFile that comes from a form input (request.FILES), since it is trying to read the file already saved in memory I suppose, and I get a TypeError on line: "stream = file_with_callback(filename, 'rb', progress.update, filename)":</p> <pre><code>coercing to Unicode: need string or buffer, InMemoryUploadedFile found </code></pre> <p>How can I achieve the same progress reporting with the user-uploaded file? Also, will this consume alot of memory reading out the progress like this, perhaps an upload solution to this <a href="https://stackoverflow.com/questions/2028517/python-urllib2-progress-hook">download progress for urllib2</a> will work better, but how to implement... Help is sooo welcome</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USHdN8
    plurals
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POPython HTTPConnection file send with httplib, retrieving progress
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POPython HTTPConnection file send with httplib, retrieving progress
      1. This table or related slice is empty.
      1. VTUpMod
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload