StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POPython: making a class method decorator that returns a method of the same class
text
Body
copied!<p>I'm new to decorators and maybe this is biting off more than I can chew for a first decorator project, but what I want to do is make a <code>parallel</code> decorator that takes a function that looks like it humbly applies to a single argument, and automatically distributes it with <code>multiprocessing</code> and turns it into a function that applies to a list of arguments.</p> <p>I am following up on <a href="https://stackoverflow.com/questions/11726809/python-efficient-workaround-for-multiprocessing-a-function-that-is-a-data-membe">this very helpful answer</a> to an earlier question, so I can successfully pickle class instance methods and I can get examples like the answer there to work just fine. </p> <p>This is my first attempt at a parallel decorator (after consulting some web hits for threading decorators).</p> <pre><code>########### # Imports # ########### import types, copy_reg, multiprocessing as mp import pandas, numpy as np ### End Imports ################## # Module methods # ################## # Parallel decorator def parallel(f): def executor(*args): _pool = mp.Pool(2) _result = _pool.map_async(f, args[1:]) # I used args[1:] because the input will be a # class instance method, so gotta skip over the self object. # but it seems like there ought to be a better way... _pool.close() _pool.join() return _result.get() return executor ### End parallel def _pickle_method(method): func_name = method.im_func.__name__ obj = method.im_self cls = method.im_class cls_name = '' if func_name.startswith('__') and not func_name.endswith('__'): cls_name = cls.__name__.lstrip('_') if cls_name: func_name = '_' + cls_name + func_name return _unpickle_method, (func_name, obj, cls) ### End _pickle_method def _unpickle_method(func_name, obj, cls): for cls in cls.mro(): try: func = cls.__dict__[func_name] except KeyError: pass else: break return func.__get__(obj, cls) ### End _unpickle_method # This call to copy_reg.pickle allows you to pass methods as the first arg to # mp.Pool methods. If you comment out this line, `pool.map(self.foo, ...)` results in # PicklingError: Can't pickle <type 'instancemethod'>: attribute lookup # __builtin__.instancemethod failed copy_reg.pickle(types.MethodType, _pickle_method, _unpickle_method) copy_reg.pickle(types.FunctionType, _pickle_method, _unpickle_method) ### End Module methods ################## # Module classes # ################## class Foo(object): def __init__(self, args): self.my_args = args ### End __init__ def squareArg(self, arg): return arg**2 ### End squareArg def par_squareArg(self): p = mp.Pool(2) # Replace 2 with the number of processors. q = p.map_async(self.squareArg, self.my_args) p.close() p.join() return q.get() ### End par_SquarArg @parallel def parSquare(self, num): return self.squareArg(num) ### End parSquare ### End Foo ### End Module classes ########### # Testing # ########### if __name__ == "__main__": myfoo = Foo([1,2,3,4]) print myfoo.par_squareArg() print myfoo.parSquare(myfoo.my_args) ### End Testing </code></pre> <p>But when I use this approach (with the silly attempt to strong arm pickling functions with the same <code>_pickle_method</code> and <code>_unpickle_method</code>) I get an error first saying, <code>AttributeError: 'function' object has no attribute 'im_func'</code> but more generally the error says that functions can't be pickled.</p> <p>So the question is twofold. (1) How could I modify the decorator so that if the <code>f</code> object it takes is an instance method of a class, then the <code>executor</code> it returns is also an instance method of that class object (so that this business about not being able to pickle does not happen, since I <em>can</em> pickle those instance methods)? and (2) Is it better to create addiitional <code>_pickle_function</code> and <code>_unpickle_function</code> methods? I thought Python could pickle module-level functions, so if my code doesn't result in <code>executor</code> becoming an instance method, it seems like it should be a module level function then, but then why can't it be pickled?</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload