Databases
StackOverflow2013
Postsdbo
POUrllib2 .read() hangs when accessing a .csv file from HTTP
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POUrllib2 .read() hangs when accessing a .csv file from HTTP
    primarykey
    data
    text
    <p>I am trying to use urllib2 to access .csv files which are posted at a specific http address. The code that I am using worked ~1 month ago but now will hang sporadically on the .read() command. I've done various tests to try to pin down the culprit. The address I am trying to access is publically available weather data from: </p> <p><a href="http://climate.weatheroffice.gc.ca/climateData/bulkdata_e.html?StationID=47267&amp;hlyRange=2008-07-15|2013-03-20&amp;timeframe=1&amp;Prov=ONT&amp;format=csv&amp;Year=2008&amp;Day=15&amp;Month=9" rel="nofollow">http://climate.weatheroffice.gc.ca/climateData/bulkdata_e.html?StationID=47267&amp;hlyRange=2008-07-15|2013-03-20&amp;timeframe=1&amp;Prov=ONT&amp;format=csv&amp;Year=2008&amp;Day=15&amp;Month=9</a></p> <p>The .readline() function works every time, and so I wrote the following test to see where the read process ends in the file: </p> <pre><code>foo=urllib2.urlopen(HTTPadress) for i in range(1000): print i,dd.readline() </code></pre> <p>This will print out each line until the program hangs. If it hangs it will be at the same line each time,and there are no special formatting or characters at this line (usually around line 680). For different datafiles, it will hang at a different line, but always the same one for the same file. </p> <p>The code that I am actually using is part of a larger function which is used to loop through multiple datafiles: </p> <pre><code>def qry(query): data = urllib2.urlopen('http://climate.weatheroffice.gc.ca/climateData/bulkdata_e.html?'+query) print 'done' #pdb.set_trace() time.sleep(5) tmp=data.read() return tmp </code></pre> <p>the sleep function between the urlopen and read functions seemed to improve reliability for a while, and this function generally works when I run it through pdb. At this point, i'm fairly convinced that something has changed on the server end which is interfering with the .read() function, but I have no idea what this is and how to work around it. </p> <p>Thanks! </p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USRob andrews
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. COI'm confused. You say "the .readline() function works every time", but then you say that it hangs at a certain line. Those can't both be true.
      singulars
      1. POUrllib2 .read() hangs when accessing a .csv file from HTTP
      1. USabarnert
    2. COYou might want to try printing out the _byte_ count, instead of the _line_ count. I'd give better than even odds that the byte count is the same for every resource, and the only reason the line count is different is that different resources with different content obviously have different-length lines.
      singulars
      1. POUrllib2 .read() hangs when accessing a .csv file from HTTP
      1. USabarnert
    3. COAlso, does it actually fail on the one example URL you gave? Because I can run that over and over again, See http://pastebin.com/1BrRQuHB. When I run this, it just prints `99929 737` once/second until I hit ^C. Does it work for you? If so, do you have a URL it _doesn't_ work for?
      singulars
      1. POUrllib2 .read() hangs when accessing a .csv file from HTTP
      1. USabarnert
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload