StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>The current state of Python's multi-processing capabilities are not great for CPU bound processing. I fear to tell you that there is no way to make it run faster using the <code>multiprocessing</code> module nor is it your use of <code>multiprocessing</code> that is the problem.</p> <p>The real problem is that Python is still bound by the rules of the <a href="http://www.dabeaz.com/GIL/" rel="nofollow">GlobalInterpreterLock(GIL)</a> (I highly suggest the <a href="http://www.dabeaz.com/python/UnderstandingGIL.pdf" rel="nofollow">slides</a>). There have been some exciting theoretical and <a href="http://mail.python.org/pipermail/pypy-dev/2012-May/009901.html" rel="nofollow">experimental advances</a> on working around the GIL. Python 3.2 event contains a new GIL which solves some of the issues, but introduces others.</p> <p>For now, it is faster to execute many Python process with a single serial thread than to attempt to run many threads within one process. This will allow you avoid issues of acquiring the GIL between threads (by effectively having more GILs). This however is only beneficial if the IPC overhead between your Python processes doesn't eclipse the benefits of the processing.</p> <p>Eli Bendersky wrote a decent <a href="http://eli.thegreenplace.net/2012/01/16/python-parallelizing-cpu-bound-tasks-with-multiprocessing/" rel="nofollow">overview article</a> on his experiences with attempting to make a CPU bound process run faster with multiprocessing.</p> <p>It is worth noting that <a href="http://python.org/dev/peps/pep-0371/" rel="nofollow">PEP 371</a> had the desire to 'side-step' the GIL with the introduction of the <code>multiprocessing</code> module (previously a non-standard packaged named <code>pyProcessing</code>). However the GIL still seems to play too large of a role in the Python interpreter to make it work well with CPU bound algorithms. Many different people have worked on removing/rewriting the GIL, but nothing has made enough traction to make it into a Python release.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload