Databases
StackOverflow2013
Postsdbo
POGevent monkeypatching breaking multiprocessing
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POGevent monkeypatching breaking multiprocessing
    primarykey
    data
    text
    <p>I am attempting to use multiprocessing's pool to run a group of processes, each of which will run a gevent pool of greenlets. The reason for this is that there is a lot of network activity, but also a lot of CPU activity, so to maximise my bandwidth and all of my CPU cores, I need multiple processes AND gevent's async monkey patching. I am using multiprocessing's manager to create a queue which the processes will access to get data to process.</p> <p>Here is a simplified fragment of the code:</p> <pre><code>import multiprocessing from gevent import monkey monkey.patch_all(thread=False) manager = multiprocessing.Manager() q = manager.Queue() </code></pre> <p>Here is the exception it produces:</p> <pre><code>Traceback (most recent call last): File "multimonkeytest.py", line 7, in &lt;module&gt; q = manager.Queue() File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/managers.py", line 667, in temp token, exp = self._create(typeid, *args, **kwds) File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/managers.py", line 565, in _create conn = self._Client(self._address, authkey=self._authkey) File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/connection.py", line 175, in Client answer_challenge(c, authkey) File "/usr/local/Cellar/python/2.7.2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/connection.py", line 409, in answer_challenge message = connection.recv_bytes(256) # reject large message IOError: [Errno 35] Resource temporarily unavailable </code></pre> <p>I believe this must be due to some difference between the behaviour of the normal socket module and gevent's socket module.</p> <p>If I monkeypatch within the subprocess, The queue is created successfully, but when the subprocess tries to get() from the queue, a very similar exception occurs. The socket does need to be monkeypatched due to doing large numbers of network requests in the subprocesses.</p> <p>My version of gevent, which I believe is the latest:</p> <pre><code>&gt;&gt;&gt; gevent.version_info (1, 0, 0, 'alpha', 3) </code></pre> <p>Any ideas?</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USuser964375
    plurals
    1. This table or related slice is empty.
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POGevent monkeypatching breaking multiprocessing
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POGevent monkeypatching breaking multiprocessing
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POGevent monkeypatching breaking multiprocessing
      1. USMatt Joiner
      1. VTFavorite
    1. COrelated: http://bugs.python.org/issue6056
      singulars
      1. POGevent monkeypatching breaking multiprocessing
      1. USjfs
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload