Databases
StackOverflow2013
Postsdbo
POParallelism with SciPy.optimize
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POParallelism with SciPy.optimize
    primarykey
    data
    text
    <p>I'm working on some research code which uses <code>scipy.optimize.leastsq</code> to optimize a function. It does this about 18 times per iteration, so I would like call leastsq in parallel to reduce running time. This shouldn't be a problem because the optimizations are almost completely separate so very little synchronization is required. I recently found out about <code>multiprocessing.pool.ThreadPool</code> which would allow me to do this without having to explicitly set up shared memory (a pain since most of my data are in NumPy arrays). So I made a slight rewrite of my code, hoping it would work, but it throws a strange error: <code>SystemError: null argument to internal routine</code>.</p> <p>The following is a simplification of my code:</p> <pre><code>def optfunc(id): def errfunc(x): return somedata[id] - somefunc(x) lock.acquire() x0 = numpy.copy(currentx[id]) lock.release() result = scipy.optimize.leastsq(errfunc, x0) lock.acquire() currentx[id] = result lock.release() ThreadPool(processes=8).map(optfunc, range(idcount)) </code></pre> <p>This should work fine, unless <code>scipy.optimize.leastsq</code> isn't threadsafe. So I tried putting a lock around <code>scipy.optimize.leastsq</code>; lo and behold it works. However, processor utilization is stuck at 100%, so this is useless to me.</p> <p>My question is then what can I do about this? I think my options are:</p> <ol> <li>Find a thread-safe implementation of LM (levmar, maybe?)</li> <li>Try using processes rather than threads (I don't think this would make a difference)</li> </ol> <p>Any help or suggestions would be greatly appreciated.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USSteve
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POParallelism with SciPy.optimize
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POParallelism with SciPy.optimize
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POParallelism with SciPy.optimize
      1. This table or related slice is empty.
      1. VTUpMod
    1. COAre you using Windows? There is a known problem with threading in optimize.leastsq on Windows: http://projects.scipy.org/scipy/ticket/1117.
      singulars
      1. POParallelism with SciPy.optimize
      1. USJustin Peel
    2. COHmm, interesting that did not come up when I searched. I'm using Ubuntu but maybe it affects both operating systems.
      singulars
      1. POParallelism with SciPy.optimize
      1. USSteve
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload