StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PONon-blocking, non-concurrent tasks in Python
text
Body
copied!<p>I am working on an implementation of a very small library in Python that <strong>has to be</strong> non-blocking.</p> <p>On some production code, at some point, a call to this library will be done and it needs to do its own work, in its most simple form it would be a callable that needs to pass some information to a service.</p> <p>This "passing information to a service" is a non-intensive task, probably sending some data to an HTTP service or something similar. It also doesn't need to be concurrent or to share information, <strong>however</strong> it does need to terminate at some point, possibly with a timeout.</p> <p>I have used the <code>threading</code> module before and it seems the most appropriate thing to use, but the application where this library will be used is so big that I am worried of hitting the threading limit. </p> <p>On local testing I was able to hit that limit at around ~2500 threads spawned.</p> <p>There is a good possibility (given the size of the application) that I can hit that limit easily. It also makes me weary of using a Queue given the memory implications of placing tasks at a high rate in it.</p> <p>I have also looked at <code>gevent</code> but I couldn't see an example of being able to spawn something that would do some work and terminate without joining. The examples I went through where calling <code>.join()</code> on a spawned <code>Greenlet</code> or on an array of greenlets.</p> <p>I don't need to know the result of the work being done! It just needs to fire off and try to talk to the HTTP service and die with a sensible timeout if it didn't.</p> <p>Have I misinterpreted the guides/tutorials for <code>gevent</code> ? Is there any other possibility to spawn a callable in fully non-blocking fashion that can't hit a ~2500 limit?</p> <p>This is a simple example in Threading that does work as I would expect:</p> <pre><code>from threading import Thread class Synchronizer(Thread): def __init__(self, number): self.number = number Thread.__init__(self) def run(self): # Simulating some work import time time.sleep(5) print self.number for i in range(4000): # totally doesn't get past 2,500 sync = Synchronizer(i) sync.setDaemon(True) sync.start() print "spawned a thread, number %s" % i </code></pre> <p>And this is what I've tried with gevent, where it obviously blocks at the end to see what the workers did:</p> <pre><code>def task(pid): """ Some non-deterministic task """ gevent.sleep(1) print('Task', pid, 'done') for i in range(100): gevent.spawn(task, i) </code></pre> <p><em>EDIT:</em> My problem stemmed out from my lack of familiarity with <code>gevent</code>. While the <code>Thread</code> code was indeed spawning threads, it also prevented the script from terminating while it did some work.</p> <p><code>gevent</code> doesn't really do that in the code above, unless you add a <code>.join()</code>. All I had to do to see the <code>gevent</code> code do some work with the spawned greenlets was to make it a long running process. This definitely fixes my problem as the code that needs to spawn the greenlets is done within a framework that is a long running process in itself.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload