StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PODeadlock in concurrent.futures code
text
Body
copied!<p>I've been trying to parallelise some code using <code>concurrent.futures.ProcessPoolExecutor</code> but have kept having strange deadlocks that don't occur with <code>ThreadPoolExecutor</code>. A minimal example:</p> <pre><code>from concurrent import futures def test(): pass with futures.ProcessPoolExecutor(4) as executor: for i in range(100): print('submitting {}'.format(i)) executor.submit(test) </code></pre> <p>In python 3.2.2 (on 64-bit Ubuntu), this seems to hang consistently after submitting all the jobs - and this seems to happen whenever the number of jobs submitted is greater than the number of workers. If I replace <code>ProcessPoolExecutor</code> with <code>ThreadPoolExecutor</code> it works flawlessly.</p> <p>As an attempt to investigate, I gave each future a callback to print the value of <code>i</code>:</p> <pre><code>from concurrent import futures def test(): pass with futures.ProcessPoolExecutor(4) as executor: for i in range(100): print('submitting {}'.format(i)) future = executor.submit(test) def callback(f): print('callback {}'.format(i)) future.add_done_callback(callback) </code></pre> <p>This just confused me even more - the value of <code>i</code> printed out by <code>callback</code> is the value at the time it is called, rather than at the time it was defined (so I never see <code>callback 0</code> but I get lots of <code>callback 99</code>s). Again, <code>ThreadPoolExecutor</code> prints out the expected value.</p> <p>Wondering if this might be a bug, I tried a recent development version of python. Now, the code at least seems to terminate, but I still get the wrong value of <code>i</code> printed out.</p> <p>So can anyone explain:</p> <ul> <li><p>what happened to <code>ProcessPoolExecutor</code> in between python 3.2 and the current dev version that apparently fixed this deadlock</p></li> <li><p>why the 'wrong' value of <code>i</code> is being printed</p></li> </ul> <p>EDIT: as jukiewicz pointed out below, of course printing <code>i</code> will print the value at the time the callback is called, I don't know what I was thinking... if I pass a callable object with the value of <code>i</code> as one of its attributes, that works as expected.</p> <p>EDIT: a little bit more information: all of the callbacks are executed, so it looks like it is <code>executor.shutdown</code> (called by <code>executor.__exit__</code>) that is unable to tell that the processes have completed. This does seem to be completely fixed in the current python 3.3, but there seem to have been a lot of changes to <code>multiprocessing</code> and <code>concurrent.futures</code>, so I don't know what fixed this. Since I can't use 3.3 (it doesn't seem to be compatible with either the release or dev versions of numpy), I tried simply copying its multiprocessing and concurrent packages across to my 3.2 installation, which seems to work fine. Still, it seems a little weird that - as far as I can see - <code>ProcessPoolExecutor</code> is completely broken in the latest release version but nobody else is affected.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload