StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PONeed help to parallelize a loop in python
text
Body
copied!<p>I have a huge data set and I have to compute for every point of it a series of properties. My code is really slow and I would like to make it faster parallelizing somehow the do loop. I would like each processor to compute the "series of properties" for a limited subsample of my data and then join all the properties together in one array. I'll try explain what I have to do with an example.</p> <p>Let's say that my data set is the array <code>x</code>:</p> <pre><code>x = linspace(0,20,10000) </code></pre> <p>The "property" I want to get is, for instance, the square root of <code>x</code>:</p> <pre><code>prop=[] for i in arange(0,len(x)): prop.append(sqrt(x[i])) </code></pre> <p>The question is how can I parallelize the above loop? Let's assume I have 4 processor and I would like each of them to compute the sqrt of 10000/4=2500 points.</p> <p>I tried looking at some python modules like <code>multiprocessing</code> and <code>mpi4py</code> but from the guides I couldn't find the answer to such a simple question.</p> <p><strong>EDITS</strong></p> <p>I'll thank you all for the precious comments and links you provided me. However, I would like to clarify my question. I'm not interested in the <code>sqrt</code> function whatsoever. I am doing a series of operations within a loop. I perfectly know loops are bad and vectorial operation are always preferable to them but in this case I really have to do a loop. I won't go into the details of my problem because this would add an unnecessary complication to this question. I would like to split my loop so that each processor does a part of it, meaning that I could run my code 40 times with 1/40 of the loop each and the merger the result but this would be stupid. This is a brief example</p> <pre><code> for i in arange(0,len(x)): # do some complicated stuff </code></pre> <p>What I want is use 40 cpus to do this:</p> <pre><code> for npcu in arange(0,40): for i in arange(len(x)/40*ncpu,len(x)/40*(ncpu+1)): # do some complicated stuff </code></pre> <p>Is that possible or not with python?</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload