Databases
StackOverflow2013
Postsdbo
POPerformance difference between urllib2 and asyncore
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POPerformance difference between urllib2 and asyncore
    primarykey
    data
    text
    <p>I have some questions about the performance of this simple python script:</p> <pre><code>import sys, urllib2, asyncore, socket, urlparse from timeit import timeit class HTTPClient(asyncore.dispatcher): def __init__(self, host, path): asyncore.dispatcher.__init__(self) self.create_socket(socket.AF_INET, socket.SOCK_STREAM) self.connect( (host, 80) ) self.buffer = 'GET %s HTTP/1.0\r\n\r\n' % path self.data = '' def handle_connect(self): pass def handle_close(self): self.close() def handle_read(self): self.data += self.recv(8192) def writable(self): return (len(self.buffer) &gt; 0) def handle_write(self): sent = self.send(self.buffer) self.buffer = self.buffer[sent:] url = 'http://pacnet.karbownicki.com/api/categories/' components = urlparse.urlparse(url) host = components.hostname or '' path = components.path def fn1(): try: response = urllib2.urlopen(url) try: return response.read() finally: response.close() except: pass def fn2(): client = HTTPClient(host, path) asyncore.loop() return client.data if sys.argv[1:]: print 'fn1:', len(fn1()) print 'fn2:', len(fn2()) time = timeit('fn1()', 'from __main__ import fn1', number=1) print 'fn1: %.8f sec/pass' % (time) time = timeit('fn2()', 'from __main__ import fn2', number=1) print 'fn2: %.8f sec/pass' % (time) </code></pre> <p>Here's the output I'm getting on linux:</p> <pre><code>$ python2 test_dl.py fn1: 5.36162281 sec/pass fn2: 0.27681994 sec/pass $ python2 test_dl.py count fn1: 11781 fn2: 11965 fn1: 0.30849886 sec/pass fn2: 0.30597305 sec/pass </code></pre> <p>Why is urllib2 so much slower than asyncore in the first run?</p> <p>And why does the discrepancy seem to disappear on the second run?</p> <p><strong>EDIT</strong>: Found a hackish solution to this problem here: <a href="https://stackoverflow.com/questions/2014534/force-python-mechanize-urllib2-to-only-use-a-requests/6319043#6319043">Force python mechanize/urllib2 to only use A requests?</a></p> <p>The five-second delay disappears if I monkey-patch the socket module as follows:</p> <pre><code>_getaddrinfo = socket.getaddrinfo def getaddrinfo(host, port, family=0, socktype=0, proto=0, flags=0): return _getaddrinfo(host, port, socket.AF_INET, socktype, proto, flags) socket.getaddrinfo = getaddrinfo </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USekhumoro
    plurals
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POPerformance difference between urllib2 and asyncore
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POPerformance difference between urllib2 and asyncore
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POPerformance difference between urllib2 and asyncore
      1. This table or related slice is empty.
      1. VTUpMod
    1. COAfter a little more research, I think I might have a lead on this. It appears that python's socket module doesn't specify the address family when creating connections. It defaults to 0 (AF_UNSPEC) rather than AF_INET (which is used in my asyncore HTTPClient class above). This can cause a 5 second delay in DNS lookup if an IPv6 response is received. The only problem with this explanation, though, is that I have IPv6 disabled on my linux box. So I'm not sure how this issue could still affect me...
      singulars
      1. POPerformance difference between urllib2 and asyncore
      1. USekhumoro
    2. COhaving it disabled doesn't mean that a client program couldn't try using it, especially when os and libraries actually have the capability.
      singulars
      1. POPerformance difference between urllib2 and asyncore
      1. USknitti
    3. CObut the kernel module isn't loaded at all, so why would python attempt to use it? i've tried re-building python with the "--disable-ipv6" flag, but that made no difference. is there anything else in my dns setup that i could change to stop this delay happening?
      singulars
      1. POPerformance difference between urllib2 and asyncore
      1. USekhumoro
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload