StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>Given the rigor you've shown with your analysis, I'm surprised by the results thus far. I put this as an 'answer' but only because it's too long for a comment and does provide a possibility (though I expect you've considered it). </p> <p>I would've thought the numpy/python approach wouldn't add much overhead for a matrix of reasonable complexity, since as the complexity increases, the proportion that python participates in should be small. I'm more interested in the results on the right hand side of the graph, but orders of magnitude discrepancy shown there would be disturbing. </p> <p>I wonder if you're using the best algorithms that numpy can leverage. From the compilation guide for linux:</p> <p>"Build FFTW (3.1.2): SciPy Versions >= 0.7 and Numpy >= 1.2: Because of license, configuration, and maintenance issues support for FFTW was removed in versions of SciPy >= 0.7 and NumPy >= 1.2. Instead now uses a built-in version of fftpack. There are a couple ways to take advantage of the speed of FFTW if necessary for your analysis. Downgrade to a Numpy/Scipy version that includes support. Install or create your own wrapper of FFTW. See <a href="http://developer.berlios.de/projects/pyfftw/" rel="noreferrer">http://developer.berlios.de/projects/pyfftw/</a> as an un-endorsed example."</p> <p>Did you compile numpy with mkl? (<a href="http://software.intel.com/en-us/articles/intel-mkl/" rel="noreferrer">http://software.intel.com/en-us/articles/intel-mkl/</a>). If you're running on linux, the instructions for compiling numpy with mkl are here: <a href="http://www.scipy.org/Installing_SciPy/Linux#head-7ce43956a69ec51c6f2cedd894a4715d5bfff974" rel="noreferrer">http://www.scipy.org/Installing_SciPy/Linux#head-7ce43956a69ec51c6f2cedd894a4715d5bfff974</a> (in spite of url). The key part is:</p> <pre><code>[mkl] library_dirs = /opt/intel/composer_xe_2011_sp1.6.233/mkl/lib/intel64 include_dirs = /opt/intel/composer_xe_2011_sp1.6.233/mkl/include mkl_libs = mkl_intel_lp64,mkl_intel_thread,mkl_core </code></pre> <p>If you're on windows, you can obtain a compiled binary with mkl, (and also obtain pyfftw, and many other related algorithms) at: <a href="http://www.lfd.uci.edu/~gohlke/pythonlibs/" rel="noreferrer">http://www.lfd.uci.edu/~gohlke/pythonlibs/</a>, with a debt of gratitude to Christoph Gohlke at the Laboratory for Fluorescence Dynamics, UC Irvine. </p> <p>Caveat, in either case, there are many licensing issues and so on to be aware of, but the intel page explains those. Again, I imagine you've considered this, but if you meet the licensing requirements (which on linux is very easy to do), this would speed up the numpy part a great deal relative to using a simple automatic build, without even FFTW. I'll be interested to follow this thread and see what others think. Regardless, excellent rigor and excellent question. Thanks for posting it.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload