StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POTwisted network client with multiprocessing workers?
text
Body
copied!<p>So, I've got an application that uses Twisted + Stomper as a STOMP client which farms out work to a multiprocessing.Pool of workers.</p> <p>This appears to work ok when I just use a python script to fire this up, which (simplified) looks something like this:</p> <pre><code># stompclient.py logging.config.fileConfig(config_path) logger = logging.getLogger(__name__) # Add observer to make Twisted log via python twisted.python.log.PythonLoggingObserver().start() # initialize the process pool. (child processes get forked off immediately) pool = multiprocessing.Pool(processes=processes) StompClientFactory.username = username StompClientFactory.password = password StompClientFactory.destination = destination reactor.connectTCP(host, port, StompClientFactory()) reactor.run() </code></pre> <p>As this gets packaged for deployment, I thought I would take advantage of the twistd script and run this from a tac file.</p> <p>Here's my very-similar-looking tac file:</p> <pre><code># stompclient.tac logging.config.fileConfig(config_path) logger = logging.getLogger(__name__) # Add observer to make Twisted log via python twisted.python.log.PythonLoggingObserver().start() # initialize the process pool. (child processes get forked off immediately) pool = multiprocessing.Pool(processes=processes) StompClientFactory.username = username StompClientFactory.password = password StompClientFactory.destination = destination application = service.Application('myapp') service = internet.TCPClient(host, port, StompClientFactory()) service.setServiceParent(application) </code></pre> <p>For the sake of illustration, I have collapsed or changed a few details; hopefully they were not the essence of the problem. For example, my app has a plugin system, the pool is initialized by a separate method, and then work is delegated to the pool using pool.apply_async() passing one of my plugin's process() methods.</p> <p>So, if I run the script (stompclient.py), everything works as expected.</p> <p>It also appears to work OK if I run twist in non-daemon mode (-n):</p> <pre><code>twistd -noy stompclient.tac </code></pre> <p>however, it does <em>not</em> work when I run in daemon mode:</p> <pre><code>twistd -oy stompclient.tac </code></pre> <p>The application appears to start up OK, but when it attempts to fork off work, it just hangs. By "hangs", I mean that it appears that the child process is never asked to do anything and the parent (that called pool.apply_async()) just sits there waiting for the response to return.</p> <p>I'm sure that I'm doing something stupid with Twisted + multiprocessing, but I'm really hoping that someone can explain to my the flaw in my approach.</p> <p>Thanks in advance!</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload