Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>There is a great module called <a href="http://docs.python.org/2/library/multiprocessing.html" rel="nofollow">multiprocessing</a> that is part of the python standard library. It will spawn processes across as many cores as you like to take advantage of the other CPUs. There is an example of using the <a href="http://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers" rel="nofollow">Pool</a> object in the docs, below is a shortened version of that example. It will calculate the square of 10 numbers distributing the work load across for work processes and displaying the result.</p> <p><strong>Simple worker pool</strong></p> <pre><code>from multiprocessing import Pool def f(x): return x*x pool = Pool(processes=4) print pool.map(f, range(10)) </code></pre> <p><strong>Output</strong></p> <pre><code>[0, 1, 4, 9, 16, 25, 36, 49, 64, 81] </code></pre> <p>I had more of a challenge to break up your problem into the same sort of structure. I had to create some intermediary functions to achieve the job. I don't have numpy so I just used lists and dictionaries in place of what you put. You can replace them and try out the code.</p> <p><strong>More involved scenario</strong> </p> <pre><code>from multiprocessing import Pool import time, pprint def fun(av, bv): time.sleep(0.1) return (av, bv) def data_stream(a, b): for i, av in enumerate(a): for j, bv in enumerate(b): yield (i, j), (av, bv) def proxy(args): return args[0], fun(*args[1]) a = range(100, 400, 100) b = range(100, 400, 100) Y = {} pool = Pool(processes=4) results = pool.map(proxy, data_stream(a, b)) for k,v in results: Y[k] = v pprint.pprint(Y) </code></pre> <p><strong>output</strong></p> <pre><code>{(0, 0): (100, 100), (0, 1): (100, 200), (0, 2): (100, 300), (1, 0): (200, 100), (1, 1): (200, 200), (1, 2): (200, 300), (2, 0): (300, 100), (2, 1): (300, 200), (2, 2): (300, 300)} </code></pre> <p><strong>Performance</strong> </p> <p>In the example I just put a dummy 0.1 second delay to simulate heavy work. But even in this example if you run a Pool with <code>processes=1</code> it runs in 0.950s with <code>processes=4</code> it runs in 0.352s. You can use the multiprocessing library in many different ways Pool is just one way. You might want to explore the examples and experiment. </p> <p>In one of the comments below there was a mention of using the chunksize argument to pool.map to help improve performance. It is important to have a general understanding of what is going on under the hood to really get a handle on performance. Basically all the data you are passing to the other processes needs to be pickled passed to the other process unpickled used and then the result goes through the same process back to the main process. There is an overhead to this inter process communication. Keep this in mind when you experiment.</p>
    singulars
    1. This table or related slice is empty.
    1. POPython multiprocessing for expensive operation with 2D array
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USMarwan Alsabbagh
    1. USMarwan Alsabbagh
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POPython multiprocessing for expensive operation with 2D array
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI also came to share about multiprocessing. It seems like the best thing for you unless you want to invest time in more in depth parallelization.
      singulars
      1. PO
      1. USJonathanV
    2. COYou might also mention increasing the `chunksize` argument to `pool.map` (or `pool.imap`). It often will make a significant difference in execution speed, particularly for cases like numpy arrays where the overhead of accessing an individual item is relatively high.
      singulars
      1. PO
      1. USJoe Kington
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload