Databases
StackOverflow2013
Postsdbo
POParallelise python loop with numpy arrays and shared-memory
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POParallelise python loop with numpy arrays and shared-memory
    primarykey
    data
    text
    <p>I am aware of several questions and answers on this topic, but haven't found a satisfactory answer to this particular problem:</p> <p>What is the easiest way to do a simple shared-memory parallelisation of a python loop where numpy arrays are manipulated through numpy/scipy functions?</p> <p>I am not looking for the most efficient way, I just wanted something simple to implement that doesn't require a significant rewrite when the loop is not run in parallel. Just like OpenMP implements in lower level languages. </p> <p>The best answer I've seen in this regard is <a href="https://stackoverflow.com/a/5550156/1580351">this one</a>, but this is a rather clunky way that requires one to express the loop into a function that takes a single argument, several lines of shared-array converting crud, seems to require that the parallel function is called from <code>__main__</code>, and it doesn't seem to work well from the interactive prompt (where I spend a lot of my time).</p> <p>With all of Python's simplicity is this really the best way to parellelise a loop? Really? This is something trivial to parallelise in OpenMP fashion.</p> <p>I have painstakingly read through the opaque documentation of the multiprocessing module, only to find out that it is so general that it seems suited to everything but a simple loop parallelisation. I am not interested in setting up Managers, Proxies, Pipes, etc. I just have a simple loop, fully parallel that doesn't have any communication between tasks. Using MPI to parallelise such a simple situation seems like overkill, not to mention it would be memory-inefficient in this case.</p> <p>I haven't had time to learn about the multitude of different shared-memory parallel packages for Python, but was wondering if someone has more experience in this and can show me a simpler way. Please do not suggest serial optimisation techniques such as Cython (I already use it), or using parallel numpy/scipy functions such as BLAS (my case is more general, and more parallel).</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. UStiago
    plurals
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    3. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POParallelise python loop with numpy arrays and shared-memory
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POParallelise python loop with numpy arrays and shared-memory
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POParallelise python loop with numpy arrays and shared-memory
      1. USdeprecated
      1. VTFavorite
    1. COrelated: [OpenMP and Python](http://stackoverflow.com/q/11368486/4279). See examples in my answer.
      singulars
      1. POParallelise python loop with numpy arrays and shared-memory
      1. USjfs
    2. COOn Linux, the code in the answer you link to works fine from the interactive prompt. Also, Cython does support openmp-based parallelization, and it is very simple to use (replace `range` by `prange` in the loop): http://docs.cython.org/src/userguide/parallelism.html
      singulars
      1. POParallelise python loop with numpy arrays and shared-memory
      1. USpv.
    3. CO@pv, thanks for the link. It looks quite simple. But I assume prange can only be used C functions? This brings other issues, such as using numpy/scipy array functions from inside Cython. I don't assume there is an easy interface for the C equivalent of those functions to be used inside Cython?
      singulars
      1. POParallelise python loop with numpy arrays and shared-memory
      1. UStiago
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload