Databases
StackOverflow2013
Postsdbo
POCan I use a multiprocessing Queue in a function called by Pool.imap?
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POCan I use a multiprocessing Queue in a function called by Pool.imap?
    text
    copied!<p>I'm using python 2.7, and trying to run some CPU heavy tasks in their own processes. I would like to be able to send messages back to the parent process to keep it informed of the current status of the process. The multiprocessing Queue seems perfect for this but I can't figure out how to get it work.</p> <p>So, this is my basic working example minus the use of a Queue.</p> <pre><code>import multiprocessing as mp import time def f(x): return x*x def main(): pool = mp.Pool() results = pool.imap_unordered(f, range(1, 6)) time.sleep(1) print str(results.next()) pool.close() pool.join() if __name__ == '__main__': main() </code></pre> <p>I've tried passing the Queue in several ways, and they get the error message "RuntimeError: Queue objects should only be shared between processes through inheritance". Here is one of the ways I tried based on an earlier answer I found. (I get the same problem trying to use Pool.map_async and Pool.imap)</p> <pre><code>import multiprocessing as mp import time def f(args): x = args[0] q = args[1] q.put(str(x)) time.sleep(0.1) return x*x def main(): q = mp.Queue() pool = mp.Pool() results = pool.imap_unordered(f, ([i, q] for i in range(1, 6))) print str(q.get()) pool.close() pool.join() if __name__ == '__main__': main() </code></pre> <p>Finally, the 0 fitness approach (make it global) doesn't generate any messages, it just locks up. </p> <pre><code>import multiprocessing as mp import time q = mp.Queue() def f(x): q.put(str(x)) return x*x def main(): pool = mp.Pool() results = pool.imap_unordered(f, range(1, 6)) time.sleep(1) print q.get() pool.close() pool.join() if __name__ == '__main__': main() </code></pre> <p>I'm aware that it will probably work with multiprocessing.Process directly and that there are other libraries to accomplish this, but I hate to back away from the standard library functions that are a great fit until I'm sure it's not just my lack of knowledge keeping me from being able to exploit them.</p> <p>Thanks.</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload