Databases
StackOverflow2013
Postsdbo
POPython Algorithm Challenge?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POPython Algorithm Challenge?
    primarykey
    data
    text
    <p>I have a <code>python</code> function (call it <code>myFunction</code>) that gets as input <strong>a list of numbers</strong>, and, following a complex calculation, returns back the result of the calculation (which is a <strong>number</strong>).</p> <p>The function looks like this:</p> <pre><code>def myFunction( listNumbers ): # initialize the result of the calculation calcResult = 0 # looping through all indices, from 0 to the last one for i in xrange(0, len(listNumbers), 1): # some complex calculation goes here, changing the value of 'calcResult' # let us now return the result of the calculation return calcResult </code></pre> <p>I tested the function, and it works as expected. </p> <p>Normally, <code>myFunction</code> is provided a <code>listNumbers</code> argument that contains <em>5,000,000</em> elements in it. As you may expect, the calculation takes time. I need this function to run as fast as possible</p> <p><strong>Here comes the challenge</strong>: assume that the time now is 5am, and that <code>listNumbers</code> contains just <em>4,999,999</em> values in it. Meaning, its LAST VALUE is <em>not yet available</em>. This value will <strong>only be available at 6am</strong>. </p> <p><em>Obviously, we can do the following</em> (<strong>1st mode</strong>): wait until <strong>6am</strong>. Then, append the last value into <code>listNumbers</code>, and then, run <code>myFunction</code>. This solution works, BUT <strong>it will take a while</strong> before <code>myFunction</code> returns our calculated result (as we need to process the <em>entire list of numbers</em>, from the first element on). Remember, <em>our goal is to get the results as soon as possible past 6am</em>.</p> <p>I was thinking about a more efficient way to solve this (<strong>2nd mode</strong>): since (at 5am) we have <code>listNumbers</code> with <em>4,999,999</em> values in it, let us immediately start running <code>myFunction</code>. Let us process <em>whatever we can</em> (remember, we don't have the last piece of data yet), and then -- exactly at 6am -- 'plug in' the new data piece -- and generate the computed result. This should be significantly faster, as <em>most of the processing</em> will be done BEFORE 6am, hence, we will only have to deal with the new data -- which means the computed result should be available <em>immediately after 6am</em>.</p> <p>Let's suppose that <strong>there's no way</strong> for us to inspect the code of <code>myFunction</code> or modify it. Is there <strong>ANY</strong> programming technique / design idea that will allow us to take <code>myFunction</code> <strong>AS IS</strong>, and do something with it (without changing its code) so that we can have it operate in the <strong>2nd mode</strong>, rather than the <strong>1st one</strong>?</p> <p>Please do not suggest using <code>c++</code> / <code>numpy + cython</code> / <code>parallel computing</code> etc to solve this problem. The goal here is to see if there's any <em>programming technique</em> or <em>design pattern</em> that can be easily used to solve such problems. </p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USuser3262424
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POPython Algorithm Challenge?
      1. This table or related slice is empty.
      1. VTDownMod
    2. VO
      singulars
      1. POPython Algorithm Challenge?
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POPython Algorithm Challenge?
      1. This table or related slice is empty.
      1. VTUpMod
    1. COObligatory unhelpful answer: Just use Haskell, so you get lazy evaluation (your 2nd mode) for free!
      singulars
      1. POPython Algorithm Challenge?
      1. USjwodder
    2. COYou say in other comments that `listNumbers` is currently a numpy array. Could you show how it's getting populated, and what causes it to wait until 6am to receive the last element? Is there threading involved?
      singulars
      1. POPython Algorithm Challenge?
      1. USKarl Knechtel
    3. COKarl: `listNumbers` loads its first *4,999,999 values* from disk (at **5am**). At **6pm**, the main program gets a message with the most recent value, and calls a method that updates the last value of `listNumbers` with the new value. No threading is involved at present.
      singulars
      1. POPython Algorithm Challenge?
      1. USuser3262424
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload