Databases
StackOverflow2013
Postsdbo
PORead large file in parallel?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PORead large file in parallel?
    primarykey
    data
    text
    <p>I have a large file which I need to read in and make a dictionary from. I would like this to be as fast as possible. However my code in python is too slow. Here is a minimal example that shows the problem.</p> <p>First make some fake data</p> <pre><code>paste &lt;(seq 20000000) &lt;(seq 2 20000001) &gt; largefile.txt </code></pre> <p>Now here is a minimal piece of python code to read it in and make a dictionary.</p> <pre><code>import sys from collections import defaultdict fin = open(sys.argv[1]) dict = defaultdict(list) for line in fin: parts = line.split() dict[parts[0]].append(parts[1]) </code></pre> <p>Timings:</p> <pre><code>time ./read.py largefile.txt real 0m55.746s </code></pre> <p>However it is possible to read the whole file much faster as:</p> <pre><code>time cut -f1 largefile.txt &gt; /dev/null real 0m1.702s </code></pre> <blockquote> <p>My CPU has 8 cores, is it possible to parallelize this program in python to speed it up?</p> </blockquote> <p>One possibility might be to read in large chunks of the input and then run 8 processes in parallel on different non-overlapping subchunks making dictionaries in parallel from the data in memory then read in another large chunk. Is this possible in python using multiprocessing somehow?</p> <p><strong>Update</strong>. The fake data was not very good as it had only one value per key. Better is</p> <pre><code>perl -E 'say int rand 1e7, $", int rand 1e4 for 1 .. 1e7' &gt; largefile.txt </code></pre> <p>(Related to <a href="https://stackoverflow.com/questions/18086424/read-in-large-file-and-make-dictionary">Read in large file and make dictionary</a> .)</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USAnush
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. PORead large file in parallel?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PORead large file in parallel?
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PORead large file in parallel?
      1. USeleanora
      1. VTBountyStart
    1. COthe list is probably slowing you down.
      singulars
      1. PORead large file in parallel?
      1. USsihrc
    2. CO@sihrc Most of the time is spent in dict[parts[0]].append(parts[1]) according to the profiling but append does take around 10% of the time. Note that in this fake case lists only have length 1 but in general I expect them to be longer.
      singulars
      1. PORead large file in parallel?
      1. USAnush
    3. COThat's what I mean. That .append part is slowing you down. Have you thought of alternatives to appending large lists? What kind of data are you appending?
      singulars
      1. PORead large file in parallel?
      1. USsihrc
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload