Databases
StackOverflow2013
Postsdbo
POProblem with multiprocessing.Pool in Python
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POProblem with multiprocessing.Pool in Python
    primarykey
    data
    text
    <p>I am having a bit of trouble with Parallel processing in Python. I am completely new to the concept of Parallel computing. I use multiprocessing that comes with standard Python.</p> <p>I have 12 threads in my computer. I ask for 12 workers, but I am not always able to get all workers I ask for. My problem arises when I don't get access to as many workers I need to process nTasks number of tasks in my code below (currently set to four). What happens then is just that the code gets stuck and never gets to what is under the comment "# Get Results". It seems random how many workers I get (I always ask for 12), but the problem arises when I get three workers or less in the following code:</p> <pre><code>import multiprocessing as mp import scipy as sp import scipy.stats as spstat import pylab def testfunc(x0, N): print 'working with x0 = %s' % x0 x = [x0] for i in xrange(1,N): x.append(spstat.norm.rvs(size = 1)) # stupid appending to make it slower if i % 10000 == 0: print 'x0 = %s, i = %s' % (x0, i) return sp.array(x) def testfuncParallel(fargs): return testfunc(*fargs) pool = mp.Pool(12) # I have 12 threads nTasks = 4 N = 100000 tasks = [(x, n) for x, n in enumerate(nTasks*[N])] # nTasks different tasks result = pool.map(testfuncParallel, tasks) pool.close() pool.join() # Get results: sim = sp.zeros((N, nTasks)) for nn, res in enumerate(result): sim[:, nn] = res pylab.figure() for i in xrange(nTasks): pylab.subplot(nTasks,1, i + 1) pylab.plot(sim[:, i]) pylab.show() </code></pre> <p>I have tried to use pool.map_async instead of pool.map but I cannot get around the problem.</p> <p>Thanks in advance,</p> <p>Sincerely,</p> <p>Matias</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USmatiasq
    plurals
    1. PL
      singulars
      1. LTLinked
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. COthis seems to work fine for me -- what versions of multiprocessing, scipy and pylab are you using?
      singulars
      1. POProblem with multiprocessing.Pool in Python
      1. USNoah
    2. COwhy are you asking for 12 processes but only submitting 4 tasks?
      singulars
      1. POProblem with multiprocessing.Pool in Python
      1. USNoah
    3. CO[also, pool.close() and pool.join() are only necessary with the map_async() version]
      singulars
      1. POProblem with multiprocessing.Pool in Python
      1. USNoah
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload