Databases
StackOverflow2013
Postsdbo
POquickly summing numpy arrays element-wise
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POquickly summing numpy arrays element-wise
    primarykey
    data
    text
    <p>Let's say I want to do an element-wise sum of a list of numpy arrays:</p> <pre><code>tosum = [rand(100,100) for n in range(10)] </code></pre> <p>I've been looking for the best way to do this. It seems like numpy.sum is awful:</p> <pre><code>timeit.timeit('sum(array(tosum), axis=0)', setup='from numpy import sum; from __main__ import tosum, array', number=10000) 75.02289700508118 timeit.timeit('sum(tosum, axis=0)', setup='from numpy import sum; from __main__ import tosum', number=10000) 78.99106407165527 </code></pre> <p>Reduce is much faster (to the tune of nearly two orders of magnitude):</p> <pre><code>timeit.timeit('reduce(add,tosum)', setup='from numpy import add; from __main__ import tosum', number=10000) 1.131795883178711 </code></pre> <p>It looks like reduce even has a meaningful lead over the non-numpy sum (note that these are for 1e6 runs rather than 1e4 for the above times):</p> <pre><code>timeit.timeit('reduce(add,tosum)', setup='from numpy import add; from __main__ import tosum', number=1000000) 109.98814797401428 timeit.timeit('sum(tosum)', setup='from __main__ import tosum', number=1000000) 125.52461504936218 </code></pre> <p>Are there other methods I should try? Can anyone explain the rankings?</p> <hr> <p><strong>Edit</strong></p> <p>numpy.sum is definitely faster if the list is turned into a numpy array first:</p> <pre><code>tosum2 = array(tosum) timeit.timeit('sum(tosum2, axis=0)', setup='from numpy import sum; from __main__ import tosum2', number=10000) 1.1545608043670654 </code></pre> <p>However, I'm only interested in doing a sum once, so turning the array into a numpy array would still incur a real performance penalty.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USlnmaurer
    1. USlnmaurer
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. COI'm guessing that `np.sum` first creates and array and then sums it which would explain it's poor performance... I'm guessing it would be the fastest if you had passed a `np.ndarray` to begin with.
      singulars
      1. POquickly summing numpy arrays element-wise
      1. USmgilson
    2. COAnd I'd expect reduce to beat `sum` by about 1/11 since it skips the `0 + tosum[0]` that is implicit in `sum`.
      singulars
      1. POquickly summing numpy arrays element-wise
      1. USmgilson
    3. COThat makes sense. I start with a bunch of separate arrays, so turning them into a numpy array first would incur the same performance penalty as having sum do it for me (since I'm only doing the sum once).
      singulars
      1. POquickly summing numpy arrays element-wise
      1. USlnmaurer
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload