Databases
StackOverflow2013
Postsdbo
POA clean, lightweight alternative to Python's twisted?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POA clean, lightweight alternative to Python's twisted?
    primarykey
    data
    text
    <p>A (long) while ago I wrote a web-spider that I multithreaded to enable concurrent requests to occur at the same time. That was in my Python youth, in the days before I knew about the <a href="http://www.dabeaz.com/python/GIL.pdf" rel="noreferrer">GIL</a> and the associated woes it creates for multithreaded code (IE, most of the time stuff just ends up serialized!)...</p> <p>I'd like to rework this code to make it more robust and perform better. There are basically two ways I could do this: I could use the new <a href="http://docs.python.org/library/multiprocessing.html" rel="noreferrer">multiprocessing module</a> in 2.6+ or I could go for a reactor / event-based model of some sort. I would rather do the later since it's far simpler and less error-prone.</p> <p>So the question relates to what framework would be best suited to my needs. The following is a list of the options I know about so far:</p> <ul> <li><a href="http://twistedmatrix.com/trac/" rel="noreferrer">Twisted</a>: The granddaddy of Python reactor frameworks: seems complex and a bit bloated however. Steep learning curve for a small task.</li> <li><a href="http://eventlet.net/" rel="noreferrer">Eventlet</a>: From the guys at <a href="http://lindenlab.com/" rel="noreferrer">lindenlab</a>. Greenlet based framework that's geared towards these kinds of tasks. I had a look at the code though and it's not too pretty: non-pep8 compliant, scattered with prints (why do people do this in a framework!?), API seems a little inconsistent.</li> <li><a href="http://code.google.com/p/pyev/" rel="noreferrer">PyEv</a>: Immature, doesn't seem to be anyone using it right now though it is based on libevent so it's got a solid backend.</li> <li><a href="http://docs.python.org/library/asyncore.html" rel="noreferrer">asyncore</a>: From the stdlib: über low-level, seems like a lot of legwork involved just to get something off the ground.</li> <li><a href="http://www.tornadoweb.org/" rel="noreferrer">tornado</a>: Though this is a server oriented product designed to server dynamic websites it does feature an <a href="http://github.com/facebook/tornado/blob/master/tornado/httpclient.py" rel="noreferrer">async HTTP client</a> and a simple <a href="http://github.com/facebook/tornado/blob/master/tornado/ioloop.py" rel="noreferrer">ioloop</a>. Looks like it could get the job done but not what it was intended for. [edit: doesn't run on Windows unfortunately, which counts it out for me - its a requirement for me to support this lame platform]</li> </ul> <p>Is there anything I have missed at all? Surely there must be a library out there that fits the sweet-spot of a simplified async networking library!</p> <p>[edit: big thanks to <a href="https://stackoverflow.com/users/177663/intgr">intgr</a> for his pointer to <a href="http://code.google.com/p/cogen/" rel="noreferrer">this page</a>. If you scroll to the bottom you will see there is a really nice list of projects that aim to tackle this task in one way or another. It seems actually that things have indeed moved on since the inception of Twisted: people now seem to favour a <a href="http://en.wikipedia.org/wiki/Coroutine" rel="noreferrer">co-routine</a> based solution rather than a traditional reactor / callback oriented one. The benefits of this approach are clearer more direct code: I've certainly found in the past, especially when working with <a href="http://www.boost.org/doc/libs/1_41_0/doc/html/boost_asio.html" rel="noreferrer">boost.asio</a> in C++ that callback based code can lead to designs that can be hard-to-follow and are relatively obscure to the untrained eye. Using co-routines allows you to write code that looks a little more synchronous at least. I guess now my task is to work out which one of these many libraries I like the look of and give it a go! Glad I asked now...]</p> <p>[edit: perhaps of interest to anyone who followed or stumbled on this this question or cares about this topic in any sense: I found a really great writeup of the current state of <a href="http://nichol.as/asynchronous-servers-in-python" rel="noreferrer">the available tools</a> for this job]</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USjkp
    plurals
    1. This table or related slice is empty.
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    3. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POA clean, lightweight alternative to Python's twisted?
      1. USamrox
      1. VTFavorite
    2. VO
      singulars
      1. POA clean, lightweight alternative to Python's twisted?
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POA clean, lightweight alternative to Python's twisted?
      1. UStosh
      1. VTFavorite
    1. COPython *is* multithreaded, it just doesn't allow two threads to run Python code concurrently.
      singulars
      1. POA clean, lightweight alternative to Python's twisted?
      1. USintgr
    2. CO@Intgr: indeed it is, so in theory since `socket` is a C module, if they are letting the GIL go before calling the underlying routine things might actually be concurrent. Even still, I think I want to go back to something single-threaded.
      singulars
      1. POA clean, lightweight alternative to Python's twisted?
      1. USjkp
    3. COI've learned much more from your question than from answers to it.
      singulars
      1. POA clean, lightweight alternative to Python's twisted?
      1. USDenis Otkidach
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload