StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>If you are on Linux (or any POSIX-compliant system), you can define this array as a global variable. <code>multiprocessing</code> is using <code>fork()</code> on Linux when it starts a new child process. A newly spawned child process automatically shares the memory with its parent as long as it does not change it (<a href="http://en.wikipedia.org/wiki/Copy-on-write" rel="noreferrer">copy-on-write</a> mechanism).</p> <p>Since you are saying "I don't need any kind of locks, since the array (actually a matrix) will be read-only" taking advantage of this behavior would be a very simple and yet extremely efficient approach: all child processes will access the same data in physical memory when reading this large numpy array.</p> <p>Don't hand your array to the <code>Process()</code> constructor, this will instruct <code>multiprocessing</code> to <code>pickle</code> the data to the child, which would be extremely inefficient or impossible in your case. On Linux, right after <code>fork()</code> the child is an exact copy of the parent using the same physical memory, so all you need to do is making sure that the Python variable 'containing' the matrix is accessible from within the <code>target</code> function that you hand over to <code>Process()</code>. This you can typically achieve with a 'global' variable.</p> <p>Example code:</p> <pre><code>from multiprocessing import Process from numpy import random global_array = random.random(10**4) def child(): print sum(global_array) def main(): processes = [Process(target=child) for _ in xrange(10)] for p in processes: p.start() for p in processes: p.join() if __name__ == "__main__": main() </code></pre> <p>On Windows -- which does not support <code>fork()</code> -- <code>multiprocessing</code> is using the win32 API call <code>CreateProcess</code>. It creates an entirely new process from any given executable. That's why on Windows one is <em>required</em> to pickle data to the child if one needs data that has been created during runtime of the parent.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload