StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>Use <a href="http://twistedmatrix.com" rel="noreferrer">twisted</a>! It makes this kind of thing absurdly easy compared to, say, using threads.</p> <pre><code>from twisted.internet import defer, reactor from twisted.web.client import getPage import time def processPage(page, url): # do somewthing here. return url, len(page) def printResults(result): for success, value in result: if success: print 'Success:', value else: print 'Failure:', value.getErrorMessage() def printDelta(_, start): delta = time.time() - start print 'ran in %0.3fs' % (delta,) return delta urls = [ 'http://www.google.com/', 'http://www.lycos.com/', 'http://www.bing.com/', 'http://www.altavista.com/', 'http://achewood.com/', ] def fetchURLs(): callbacks = [] for url in urls: d = getPage(url) d.addCallback(processPage, url) callbacks.append(d) callbacks = defer.DeferredList(callbacks) callbacks.addCallback(printResults) return callbacks @defer.inlineCallbacks def main(): times = [] for x in xrange(5): d = fetchURLs() d.addCallback(printDelta, time.time()) times.append((yield d)) print 'avg time: %0.3fs' % (sum(times) / len(times),) reactor.callWhenRunning(main) reactor.run() </code></pre> <p>This code also performs better than any of the other solutions posted (edited after I closed some things that were using a lot of bandwidth):</p> <pre><code>Success: ('http://www.google.com/', 8135) Success: ('http://www.lycos.com/', 29996) Success: ('http://www.bing.com/', 28611) Success: ('http://www.altavista.com/', 8378) Success: ('http://achewood.com/', 15043) ran in 0.518s Success: ('http://www.google.com/', 8135) Success: ('http://www.lycos.com/', 30349) Success: ('http://www.bing.com/', 28611) Success: ('http://www.altavista.com/', 8378) Success: ('http://achewood.com/', 15043) ran in 0.461s Success: ('http://www.google.com/', 8135) Success: ('http://www.lycos.com/', 30033) Success: ('http://www.bing.com/', 28611) Success: ('http://www.altavista.com/', 8378) Success: ('http://achewood.com/', 15043) ran in 0.435s Success: ('http://www.google.com/', 8117) Success: ('http://www.lycos.com/', 30349) Success: ('http://www.bing.com/', 28611) Success: ('http://www.altavista.com/', 8378) Success: ('http://achewood.com/', 15043) ran in 0.449s Success: ('http://www.google.com/', 8135) Success: ('http://www.lycos.com/', 30349) Success: ('http://www.bing.com/', 28611) Success: ('http://www.altavista.com/', 8378) Success: ('http://achewood.com/', 15043) ran in 0.547s avg time: 0.482s </code></pre> <p>And using Nick T's code, rigged up to also give the average of five and show the output better: </p> <pre><code>Starting threaded reads: ...took 1.921520 seconds ([8117, 30070, 15043, 8386, 28611]) Starting threaded reads: ...took 1.779461 seconds ([8135, 15043, 8386, 30349, 28611]) Starting threaded reads: ...took 1.756968 seconds ([8135, 8386, 15043, 30349, 28611]) Starting threaded reads: ...took 1.762956 seconds ([8386, 8135, 15043, 29996, 28611]) Starting threaded reads: ...took 1.654377 seconds ([8117, 30349, 15043, 8386, 28611]) avg time: 1.775s Starting sequential reads: ...took 1.389803 seconds ([8135, 30147, 28611, 8386, 15043]) Starting sequential reads: ...took 1.457451 seconds ([8135, 30051, 28611, 8386, 15043]) Starting sequential reads: ...took 1.432214 seconds ([8135, 29996, 28611, 8386, 15043]) Starting sequential reads: ...took 1.447866 seconds ([8117, 30028, 28611, 8386, 15043]) Starting sequential reads: ...took 1.468946 seconds ([8153, 30051, 28611, 8386, 15043]) avg time: 1.439s </code></pre> <p>And using Wai Yip Tung's code:</p> <pre><code>Fetched 8117 from http://www.google.com/ Fetched 28611 from http://www.bing.com/ Fetched 8386 from http://www.altavista.com/ Fetched 30051 from http://www.lycos.com/ Fetched 15043 from http://achewood.com/ done in 0.704s Fetched 8117 from http://www.google.com/ Fetched 28611 from http://www.bing.com/ Fetched 8386 from http://www.altavista.com/ Fetched 30114 from http://www.lycos.com/ Fetched 15043 from http://achewood.com/ done in 0.845s Fetched 8153 from http://www.google.com/ Fetched 28611 from http://www.bing.com/ Fetched 8386 from http://www.altavista.com/ Fetched 30070 from http://www.lycos.com/ Fetched 15043 from http://achewood.com/ done in 0.689s Fetched 8117 from http://www.google.com/ Fetched 28611 from http://www.bing.com/ Fetched 8386 from http://www.altavista.com/ Fetched 30114 from http://www.lycos.com/ Fetched 15043 from http://achewood.com/ done in 0.647s Fetched 8135 from http://www.google.com/ Fetched 28611 from http://www.bing.com/ Fetched 8386 from http://www.altavista.com/ Fetched 30349 from http://www.lycos.com/ Fetched 15043 from http://achewood.com/ done in 0.693s avg time: 0.715s </code></pre> <p>I've gotta say, I do like that the sequential fetches performed <em>better</em> for me.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload