Databases
StackOverflow2013
Postsdbo
POPython multiprocessing.starmap Process Count & Speed
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POPython multiprocessing.starmap Process Count & Speed
    primarykey
    data
    text
    <p>I have a genetic algorithm whose fitness function is a cumbersome simulation environment. The code is absolutely CPU-bound, runs on 64bit python 3.3, so I have implemented <code>multiprocessing.starmap_async</code> to parallelize.</p> <p>And it works very well, large increases in efficiency over serial. On my processor,Intel i7 CPU @ 2.40 GHz (with 16GB RAM): I notice run-times of 8 to 9 seconds with 4 processes (and slower for 2 processes, and even slower for serial).</p> <p>HOWEVER, this only utilizes 65 to 73% of my processor.<br> Increasing the process count to 6 utilizes 95% to 100% of the processor, but with a runtime of 11 seconds. Memory is still sitting around 20% </p> <p>Increase this count to 8 and the processor sits constantly at 100%, but runtime is now 12 seconds. Memory is just fine.</p> <p>I can't post everything, but below is the multiprocessing call (with the arguments removed). Is there anything I can do to utilize more of the processor without a slow-down? I'd also appreciate any help understanding why this phenomena is occurring. </p> <p>Multiprocessing call:</p> <pre><code> step = np.int8(pop_size/4) pol = Pool(processes=4) res = pol.starmap_async(SimWorker, ((i, i+step,pop,"a bunch of other arguments") for i in range(0, pop_size, step))) fitnessdict = res.get() fitnessdict = np.asarray(fitnessdict) for i in range(0,pop_size,step): for p in range(i,i+step): fitness[p] = fitnessdict[i/step,p] </code></pre> <p>SimWorker:</p> <pre><code>def SimWorker(start, stop, pop, "a bunch of other arguments"): """ Run a batch of sims """ for p in range(start, stop): fitness[p] = Sim_T(pop[p],"a bunch of other arguments") return(fitness) </code></pre>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USunutbu
    1. USDergs McGreggin
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. POPython multiprocessing.starmap Process Count & Speed
      1. USunutbu
      1. VTFavorite
    2. VO
      singulars
      1. POPython multiprocessing.starmap Process Count & Speed
      1. USusethedeathstar
      1. VTFavorite
    1. COIf you have hyperthreading (sounds like you do), the extra virtual threads will compete for the same physical CPU core. So on an i7 with 4 physical cores and 8 virtual cores, when you run a perfectly CPU-bound task with more than 4 cores, you incur overhead due to that competition.
      singulars
      1. POPython multiprocessing.starmap Process Count & Speed
      1. USmdscruggs
    2. COYes I have hyperthreading (sorry should've mentioned that). I was expecting results similar to: http://eli.thegreenplace.net/2012/01/16/python-parallelizing-cpu-bound-tasks-with-multiprocessing/ Is this a problem of Windows, and a different OS would more efficiently handle this?
      singulars
      1. POPython multiprocessing.starmap Process Count & Speed
      1. USDergs McGreggin
    3. COWhat happens if you increase the `step` size, thereby making each `SimWorker` do more work before ending?
      singulars
      1. POPython multiprocessing.starmap Process Count & Speed
      1. USunutbu
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload