StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POpython pool.map vs map
text
Body
copied!<p>I need to apply a complex function to a long series. Done sequentially this seems to be inefficient as I have access to a 12 core machine.</p> <p>Before making significant investment in this, i wrote up this simple version which compares pool.map and map. </p> <p>1) Surprisingly map is much much faster (see results below). </p> <p>2) And there is an overflow error in the pool function which does not come up on the map version.</p> <p>3) with a smaller array, the run time warning does not appear, but map is still faster.</p> <p>I am not a computer science guy (just a functional user) - any thoughts and suggestions? I went with pool.map because the async version may mess up the order of the series (which is a pain for me to sort out). </p> <p>SEE UPDATE BELOW : Based on Jdog's suggestion.</p> <p>Config: python v 2.7, 64 bit version, windows 7 </p> <pre><code>#------------------------------------------------------------------------------- # Name: poolMap #------------------------------------------------------------------------------- import multiprocessing as mp import numpy as np import time def func(x): y=x*x return y def worker(inputs): num=mp.cpu_count() print 'num of cpus', num pool = mp.Pool(num) #inputs = list(inputs) #print "inputs type",type(inputs) results = pool.map(func, inputs) pool.close() pool.join() return results if __name__ == '__main__': series = np.arange(500000) start = time.clock() poolAnswer = worker(series) end = time.clock() print 'pool time' ,(end - start) start = time.clock() answer = map(func,series) end = time.clock() print 'map time', (end - start) </code></pre> <p>results:</p> <p>num of cpus 12</p> <p>pool time 2.40276007188</p> <p>D:\poolmap.py:19: RuntimeWarning: overflow encountered in long_scalars y=x*x</p> <p>map time 0.904187849745</p> ##############UPDATE <p>Using this func gave me the results I was looking for</p> <pre><code>def func(x): x=float(x) y=(((x*x)**0.35))*x+np.ma.sqrt((((x*x)**0.35))) return y </code></pre> <p>results: num of cpus 12</p> <p>pool time 12.7410957475</p> <p>map time 45.4550067581</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload