Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>Definitely do the profiler thing but... I think the only major speedup will happen due to parallelism. Taking advantage of multiple cores is of paramount importance if you are going to run cpu bound problems like this. Try putting each line through a different (thread/process). This raises more questions of course, for example does the data need to be in the same order from the input file? if so just enumerate that and stick a second variable on the big_hairy_func for which line it will be.</p> <p>here is some boilerplate code to get started</p> <p>notes:</p> <p>xreadlines is deprecated even though it deals with large files <code>for line in file:</code> replaces it.</p> <pre><code>fi = open('1.txt') fo = open('2.txt','w') import math def log(x): return math.log(x) from math import sqrt import multiprocessing as mp import sys sys.path.append('/tools/lib/python2.7/site-packages') import scipy import numpy as np from scipy.stats.distributions import norm def big_hairy_func(linefromfile): &lt;majority of your post here&gt; return &lt;whatever data you were going to write to 'fo'&gt; if __name__ == '__main__': pool = mp.Pool(4) #rule of thumb. Replace '4' with the number of cores on your system result = pool.map(big_hairy_func, (input for input in fi.readlines())) &lt;write the result to fo that you haven't posted&gt; </code></pre> <p>xreadlines was deprecated in python 2.3 so with that version I'm not sure if the generator function will work. Let me know if you have questions about compatibility with your version of python.</p>
    singulars
    1. This table or related slice is empty.
    1. POPython: How to reduce run time?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USBack2Basics
    1. USBack2Basics
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTDownMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. CO(-1) Good advice in general, a bad one in this particular case. `numpy` as a dependency already, the code seems vectorizeable; hence a speedup by at least a factor of a few (possibly a few hundred) is nearly for free. Starting from multiprocessing seems to be an overkill here.
      singulars
      1. PO
      1. USev-br
    2. COAlthough numpy is imported it's not used.
      singulars
      1. PO
      1. USBack2Basics
    3. COyes, and that's the first optimization to throw in, to use it. If the vectorized code is still not fast enough, then there's multiprocessing. So, I'd be +1 (or +more) on your suggestion *if and when* it's vectorized.
      singulars
      1. PO
      1. USev-br
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload