StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POCorrect way to send a POST request with Python (urllib2 , base64 and json)
text
Body
copied!<p>I need to send thousands of POST requests. I searched information on the web and after many tries I coded the following solution:</p> <pre><code>post_data = urllib.urlencode({'pushes': b64encode(json.dumps(pushes))}) user_agent = 'Mozilla/4.0 (compatible; MSIE 5.5; Windows NT)' headers = { 'User-Agent' : user_agent, 'Content-type': "application/x-www-form-urlencoded", 'Accept': "text/plain"} request = urllib2.Request(PUSH_URL, data=post_data, headers=headers) response = urllib2.urlopen(request).read() </code></pre> <p>pushes is a list of dictionaries.</p> <p>Most of times this solution works fine, but sometimes it causes a segmentation fault on destination server (Apache). I have monitored the opened connections on destination server. The number of connections is low (100-200). Perhaps my code is not 100% correct. I need a more reliable solution.</p> <p>Thanks in advance.</p> <p><strong>UPDATE - Jan 25 2013</strong><br /> I used strace to debug the apache web server (destination server).<br/> I found out that there is some problem with the syscall 'gettimeofday' which is used by the Zend Framework.<br/> I tried to moved the zend application in two different servers, Centos and Ubuntu Server 12.04.<br /> The PHP version was changed from 5.3 to 5.4 too.<br /> I always got the same problem with strace.</p> <p>I ported the application to another framework. The problem is vanished.<br/> The code on the source server which does the POST requests was not modified.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload