Databases
StackOverflow2013
Postsdbo
POHow to speed up this Python code?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow to speed up this Python code?
    primarykey
    data
    text
    <p>I've got the following tiny Python method that is <strong><em>by far</em></strong> the performance hotspot (according to my profiler, >95% of execution time is spent here) in a much larger program:</p> <pre><code>def topScore(self, seq): ret = -1e9999 logProbs = self.logProbs # save indirection l = len(logProbs) for i in xrange(len(seq) - l + 1): score = 0.0 for j in xrange(l): score += logProbs[j][seq[j + i]] ret = max(ret, score) return ret </code></pre> <p>The code is being run in the Jython implementation of Python, not CPython, if that matters. <code>seq</code> is a DNA sequence string, on the order of 1,000 elements. <code>logProbs</code> is a list of dictionaries, one for each position. The goal is to find the maximum score of any length <code>l</code> (on the order of 10-20 elements) subsequence of <code>seq</code>.</p> <p>I realize all this looping is inefficient due to interpretation overhead and would be a heck of a lot faster in a statically compiled/JIT'd language. However, I'm not willing to switch languages. First, I need a JVM language for the libraries I'm using, and this kind of constrains my choices. Secondly, I don't want to translate this code wholesale into a lower-level JVM language. However, I'm willing to rewrite this hotspot in something else if necessary, though I have no clue how to interface it or what the overhead would be.</p> <p>In addition to the single-threaded slowness of this method, I also can't get the program to scale much past 4 CPUs in terms of parallelization. Given that it spends almost all its time in the 10-line hotspot I've posted, I can't figure out what the bottleneck could be here.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USdsimcha
    plurals
    1. PL
      singulars
      1. LTLinked
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POHow to speed up this Python code?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POHow to speed up this Python code?
      1. USascobol
      1. VTFavorite
    3. VO
      singulars
      1. POHow to speed up this Python code?
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI can't quite get my head around the data structure you are using. Could you post a shortened sample of `seq` and `logProbs`?
      singulars
      1. POHow to speed up this Python code?
      1. USKatriel
    2. COMy first thought was numpy, so maybe something on this page might be of use: http://stackoverflow.com/questions/316410/is-there-a-good-numpy-clone-for-jython
      singulars
      1. POHow to speed up this Python code?
      1. USRussell Borogove
    3. COMy second thought is everting the iteration such that you go over seq only one time, but that probably means that logProbs and score become more complex, and may not actually reduce work done.
      singulars
      1. POHow to speed up this Python code?
      1. USRussell Borogove
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload