StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POWhy is numpy.array() is sometimes very slow?
text
Body
copied!<p>I'm using the numpy.array() function to create numpy.float64 ndarrays from lists.</p> <p>I noticed that this is very slow when either the list contains None or a list of lists is provided.</p> <p>Below are some examples with times. There are obvious workarounds but why is this so slow?</p> <p>Examples for list of None:</p> <pre><code>### Very slow to call array() with list of None In [3]: %timeit numpy.array([None]*100000, dtype=numpy.float64) 1 loops, best of 3: 240 ms per loop ### Problem doesn't exist with array of zeroes In [4]: %timeit numpy.array([0.0]*100000, dtype=numpy.float64) 100 loops, best of 3: 9.94 ms per loop ### Also fast if we use dtype=object and convert to float64 In [5]: %timeit numpy.array([None]*100000, dtype=numpy.object).astype(numpy.float64) 100 loops, best of 3: 4.92 ms per loop ### Also fast if we use fromiter() insead of array() In [6]: %timeit numpy.fromiter([None]*100000, dtype=numpy.float64) 100 loops, best of 3: 3.29 ms per loop </code></pre> <p>Examples for list of lists:</p> <pre><code>### Very slow to create column matrix In [7]: %timeit numpy.array([[0.0]]*100000, dtype=numpy.float64) 1 loops, best of 3: 353 ms per loop ### No problem to create column vector and reshape In [8]: %timeit numpy.array([0.0]*100000, dtype=numpy.float64).reshape((-1,1)) 100 loops, best of 3: 10 ms per loop ### Can use itertools to flatten input lists In [9]: %timeit numpy.fromiter(itertools.chain.from_iterable([[0.0]]*100000),dtype=numpy.float64).reshape((-1,1)) 100 loops, best of 3: 9.65 ms per loop </code></pre>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload