StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POHow to use the initializer in a Python ThreadPool
text
Body
copied!<p>I am trying to do threaded convolution using PyFFTW, in order to calculate a large number of 2D convolutions simultaneously. (One does not need separate processes, since the GIL is released for Numpy operations). Now here is the canonical model for doing so: <a href="http://code.activestate.com/recipes/577187-python-thread-pool/" rel="nofollow">http://code.activestate.com/recipes/577187-python-thread-pool/</a></p> <p>(Py)FFTW is so fast because it reuses plans. These have to be setup separately for each thread in order to avoid access violation errors, like this:</p> <pre><code>class Worker(Thread): """Thread executing tasks from a given tasks queue""" def __init__(self, tasks): Thread.__init__(self) self.tasks = tasks self.daemon = True # Make separate fftw plans for each thread. flag_for_fftw='patient' self.inputa = np.zeros(someshape, dtype='float32') self.outputa = np.zeros(someshape_semi, dtype='complex64') # create a forward plan. self.fft = fftw3.Plan(self.inputa,self.outputa, direction='forward', flags=[flag_for_fftw],nthreads=1) # Initialize the arrays for the inverse fft. self.inputb = np.zeros(someshape_semi, dtype='complex64') self.outputb = np.zeros(someshape, dtype='float32') # Create the backward plan. self.ifft = fftw3.Plan(self.inputb,self.outputb, direction='backward', flags=[flag_for_fftw],nthreads=1) self.start() </code></pre> <p>In this way one can pass the arguments <code>self.inputa</code>, <code>self.outputa</code>, <code>self.fft</code>, <code>self.inputb</code>, <code>self.outputb</code>, <code>self.ifft</code> to the actual convolver within the run method in the Worker class.</p> <p>This is all nice, but we might as well import the ThreadPool class:</p> <pre><code>from multiprocessing.pool import ThreadPool </code></pre> <p>But how should I define the initializer in ThreadPool to get the same result? According to the docs <a href="http://docs.python.org/library/multiprocessing.html" rel="nofollow">http://docs.python.org/library/multiprocessing.html</a> "each worker process will call initializer(*initargs) when it starts". You can easily check this in the Python source code.</p> <p>However, when you set up the Threadpool, for example with 2 threads:</p> <pre><code>po = ThreadPool(2,initializer=tobedetermined) </code></pre> <p>and you run it, perhaps in some loop</p> <pre><code>po.apply_async(convolver,(some_input,)) </code></pre> <p>how can you make convolver be setup by initializer? How can you make it use separate FFTW plans in each thread, without recomputing the FFTW plan for every convolution?</p> <p>Cheers, Alex.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload