Databases
StackOverflow2013
Postsdbo
POIs it possible to loop over an httplib.HTTPResponse's data?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POIs it possible to loop over an httplib.HTTPResponse's data?
    primarykey
    data
    text
    <p>I'm trying to develop a very simple proof-of-concept to retrieve and process data in a streaming manner. The server I'm requesting from will send data in chunks, which is good, but I'm having issues using <code>httplib</code> to iterate through the chunks.</p> <p>Here's what I'm trying: import httplib</p> <pre><code>def getData(src): d = src.read(1024) while d and len(d) &gt; 0: yield d d = src.read(1024) if __name__ == "__main__": con = httplib.HTTPSConnection('example.com', port='8443', cert_file='...', key_file='...') con.putrequest('GET', '/path/to/resource') response = con.getresponse() for s in getData(response): print s raw_input() # Just to give me a moment to examine each packet </code></pre> <p>Pretty simple. Just open an HTTPS connection to server, request a resource, and grab the result, 1024 bytes at a time. I'm definitely making the HTTPS connection successfully, so that's not a problem at all.</p> <p>However, what I'm finding is that the call to <code>src.read(1024)</code> returns the same thing every time. It only ever returns the first 1024 bytes of the response, apparently never keeping track of a cursor within the file.</p> <p>So how am I supposed to receive 1024 bytes at a time? The documentation on <code>read()</code> is pretty sparse. I've thought about using urllib or urllib2, but neither seems to be able to make an HTTPS connection.</p> <p>HTTPS is required, and I am working in a rather restricted corporate environment where packages like <a href="http://docs.python-requests.org/en/latest/" rel="nofollow">Requests</a> are a bit tough to get my hands on. If possible, I'd like to find a solution within Python's standard lib.</p> <h1>// Big Old Fat Edit</h1> <p>Turns out in my original code I had simply forgot to update the <code>d</code> variable. I initialized it with a read outside the <code>yield</code> loop and never changed it in the loop. Once I added it back in there it worked perfectly.</p> <p>So, in short, I'm just a big idiot.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USKen Bellows
    1. USKen Bellows
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. COwith urllib2, have you seen HTTPSHandler? `opener = urllib2.OpenerDirector() opener.add_handler(urllib2.HTTPSHandler())`
      singulars
      1. POIs it possible to loop over an httplib.HTTPResponse's data?
      1. USHut8
    2. COYou need a con.send() after putrequest() to get anything back. Your sample iterator works for me on HTTPConnection objects.
      singulars
      1. POIs it possible to loop over an httplib.HTTPResponse's data?
      1. UStdelaney
    3. CO@tdelaney I'm getting data back without the `con.send()`, and it is the correct data, but I'm only getting the first 1024 bits. I tried adding `con.send("")` after the `con.putrequest()` and I'm still getting the same result.
      singulars
      1. POIs it possible to loop over an httplib.HTTPResponse's data?
      1. USKen Bellows
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload