Databases
StackOverflow2013
Postsdbo
POEfficient Python way to process two huge files?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POEfficient Python way to process two huge files?
    primarykey
    data
    text
    <p>I am working on a problem where I have to find if a number falls within a certain range. However, the problem is complicated due to the fact that the files I am dealing with have hundreds of thousands of lines. </p> <p>Below I try to explain the problem in as simple a language as possible. </p> <p>Here is a brief description of my input files :</p> <p>File <strong>Ranges.txt</strong> has some ranges whose min and max are tab separated.</p> <pre><code>10 20 30 40 60 70 </code></pre> <p>This can have about 10,000,000 such lines with ranges. </p> <p><strong>NOTE:</strong> The ranges <strong>never</strong> overlap. </p> <p>File <strong>Numbers.txt</strong> has a list of numbers and some values associated with each number. </p> <pre><code>12 0.34 22 0.14 34 0.79 37 0.87 </code></pre> <p>And so on. Again there are hundreds of thousands of such lines with numbers and their associated values.</p> <p>What I wish to do is take every number from <strong>Numbers.txt</strong> and check if it falls within any of the ranges in <strong>Ranges.txt</strong>. </p> <p>For all such numbers that fall within a range, I have to get a mean of their associated values (ie a mean per range). </p> <p>For eg. in the example above in <strong>Numbers.txt</strong>, there are two numbers 34 and 37 that fall within the range 30-40 in <strong>Ranges.txt</strong>, so for the range 30-40 I have to calculate the mean of the associated values of 34 and 37. (i.e mean of 0.79 and 0.87), which is 0.82</p> <p>My final output file should be the <strong>Ranges.txt</strong> but with the mean of the associated values of all numbers falling within each range. Something like :</p> <p><strong>Output.txt</strong></p> <pre><code>10 20 &lt;mean&gt; 30 40 0.82 60 70 &lt;mean&gt; </code></pre> <p>and so on.</p> <p>Would appreciate any help and ideas on how this can be written efficiently in Python. </p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USuser1691717
    1. USuser1691717
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POEfficient Python way to process two huge files?
      1. USTalvalin
      1. VTFavorite
    2. VO
      singulars
      1. POEfficient Python way to process two huge files?
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POEfficient Python way to process two huge files?
      1. This table or related slice is empty.
      1. VTUpMod
    1. COWhat have you tried? I would think you should open the files, convert them into lists, parse them, then output to output.txt.
      singulars
      1. POEfficient Python way to process two huge files?
      1. USbozdoz
    2. COI have tried doing that, however that is slow since there are millions of such lines in each of the two files. I also tried using "continue" when a number is found to fall within a given range so that other numbers are skipped and the process becomes faster, however that is still very slow.
      singulars
      1. POEfficient Python way to process two huge files?
      1. USuser1691717
    3. COAre both files sorted? Can the ranges overlap? If they can, then what is the desired result? Merge ranges first? Assign the number to all the ranges that it occurs in?
      singulars
      1. POEfficient Python way to process two huge files?
      1. USAlok--
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload