StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>Instance methods are not automatically picklable. So </p> <pre><code>p.imap(f.f, jobs) </code></pre> <p>fails because <code>p.imap</code> tries to pickle the arguments. There is a way to "teach" pickle <a href="http://bytes.com/topic/python/answers/552476-why-cant-you-pickle-instancemethods" rel="nofollow">how to pickle instance methods</a> (see Steven Bethard's answer), but your code has another problem: Passing the queue to the instance leads to a RuntimeError:</p> <pre><code>RuntimeError: Queue objects should only be shared between processes through inheritance </code></pre> <p>The error message is a little confusing (at least to me) since you <em>can</em> pass the queue as an argument to <code>p.imap</code>, but you can not pass it to the class <code>F</code> first, and then transfer it to the worker processes through <code>f.f</code>. </p> <p>Anyway, because of these problems, I'd suggest sticking with the original code instead of trying to wrap the code in a class.</p> <hr> <p>Here is an example of how to pickle instance methods:</p> <pre><code>import multiprocessing as mp import copy_reg import types def _pickle_method(method): # Author: Steven Bethard (author of argparse) # http://bytes.com/topic/python/answers/552476-why-cant-you-pickle- # instancemethods func_name = method.im_func.__name__ obj = method.im_self cls = method.im_class cls_name = '' if func_name.startswith('__') and not func_name.endswith('__'): cls_name = cls.__name__.lstrip('_') if cls_name: func_name = '_' + cls_name + func_name return _unpickle_method, (func_name, obj, cls) def _unpickle_method(func_name, obj, cls): # Author: Steven Bethard # http://bytes.com/topic/python/answers/552476-why-cant-you-pickle- # instancemethods for cls in cls.mro(): try: func = cls.__dict__[func_name] except KeyError: pass else: break return func.__get__(obj, cls) # This call to copy_reg.pickle allows you to pass methods as the first arg to # mp.Pool methods. If you comment out this line, `pool.map(self.foo, ...)` results in # PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup # __builtin__.instancemethod failed copy_reg.pickle(types.MethodType, _pickle_method, _unpickle_method) class F(object): def f(self, x): fq.put('Doing: ' + str(x)) return x*x def f_init(q): # http://stackoverflow.com/a/3843313/190597 (Olson) global fq fq = q def main(): jobs = range(1,6) q = mp.Queue() p = mp.Pool(None, f_init, [q]) f = F() results = p.imap(f.f, jobs) p.close() for r in results: print(r, q.get()) if __name__ == '__main__': main() </code></pre> <p>yields</p> <pre><code>(1, 'Doing: 2') (4, 'Doing: 3') (9, 'Doing: 4') (16, 'Doing: 1') (25, 'Doing: 5') </code></pre>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload