Databases
StackOverflow2013
Postsdbo
POAnnoying Twisted Python problem
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POAnnoying Twisted Python problem
    primarykey
    data
    text
    <p>I'm trying to answer the following question out of personal interest: <a href="https://stackoverflow.com/questions/2632520/what-is-the-fastest-way-to-send-100-000-http-requests-in-python">What is the fastest way to send 100,000 HTTP requests in Python?</a></p> <p>And this is what I have came up so far, but I'm experiencing something very stange.</p> <p>When <strong>installSignalHandlers</strong> is <strong>True</strong>, it just hangs. I can see that the <code>DelayedCall</code> instances are in <code>reactor._newTimedCalls</code>, but <code>processResponse</code> never gets called.</p> <p>When <strong>installSignalHandlers</strong> is <strong>False</strong>, it throws an error and works.</p> <pre><code>from twisted.internet import reactor from twisted.web.client import Agent from threading import Semaphore, Thread import time concurrent = 100 s = Semaphore(concurrent) reactor.suggestThreadPoolSize(concurrent) t=Thread( target=reactor.run, kwargs={'installSignalHandlers':True}) t.daemon=True t.start() agent = Agent(reactor) def processResponse(response,url): print response.code, url s.release() def processError(response,url): print "error", url s.release() def addTask(url): req = agent.request('HEAD', url) req.addCallback(processResponse, url) req.addErrback(processError, url) for url in open('urllist.txt'): addTask(url.strip()) s.acquire() while s._Semaphore__value!=concurrent: time.sleep(0.1) reactor.stop() </code></pre> <p>And here is the error that it throws when installSignalHandlers is True: (Note: This is the expected behaviour! The question is why it doesn't work when installSignalHandlers is False.)</p> <pre><code>Traceback (most recent call last): File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 396, in fireEvent DeferredList(beforeResults).addCallback(self._continueFiring) File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 224, in addCallback callbackKeywords=kw) File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 213, in addCallbacks self._runCallbacks() File "/usr/lib/python2.6/dist-packages/twisted/internet/defer.py", line 371, in _runCallbacks self.result = callback(self.result, *args, **kw) --- &lt;exception caught here&gt; --- File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 409, in _continueFiring callable(*args, **kwargs) File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1165, in _reallyStartRunning self._handleSignals() File "/usr/lib/python2.6/dist-packages/twisted/internet/base.py", line 1105, in _handleSignals signal.signal(signal.SIGINT, self.sigInt) exceptions.ValueError: signal only works in main thread </code></pre> <p><strong>What am I doing wrong and what is the right way?</strong> I'm new to twisted.</p> <p>@moshez: Thanks. It works now:</p> <pre><code>from twisted.internet import reactor, threads from urlparse import urlparse import httplib import itertools concurrent = 100 finished=itertools.count(1) reactor.suggestThreadPoolSize(concurrent) def getStatus(ourl): url = urlparse(ourl) conn = httplib.HTTPConnection(url.netloc) conn.request("HEAD", url.path) res = conn.getresponse() return res.status def processResponse(response,url): print response, url processedOne() def processError(error,url): print "error", url#, error processedOne() def processedOne(): if finished.next()==added: reactor.stop() def addTask(url): req = threads.deferToThread(getStatus, url) req.addCallback(processResponse, url) req.addErrback(processError, url) added=0 for url in open('urllist.txt'): added+=1 addTask(url.strip()) try: reactor.run() except KeyboardInterrupt: reactor.stop() </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USTarnay Kálmán
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POAnnoying Twisted Python problem
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POAnnoying Twisted Python problem
      1. USTg.
      1. VTFavorite
    3. VO
      singulars
      1. POAnnoying Twisted Python problem
      1. This table or related slice is empty.
      1. VTUpMod
    1. COThere's no reason to handle KeyboardInterrupt being raised from reactor.run(). C-c causes reactor.run() to *return*, not to raise an exception.
      singulars
      1. POAnnoying Twisted Python problem
      1. USJean-Paul Calderone
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload