Databases
StackOverflow2013
Postsdbo
POHow can I speed up line by line reading of an ASCII file? (C++)
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow can I speed up line by line reading of an ASCII file? (C++)
    primarykey
    data
    text
    <p>Here's a bit of code that is a considerable bottleneck after doing some measuring:</p> <pre><code>//----------------------------------------------------------------------------- // Construct dictionary hash set from dictionary file //----------------------------------------------------------------------------- void constructDictionary(unordered_set&lt;string&gt; &amp;dict) { ifstream wordListFile; wordListFile.open("dictionary.txt"); std::string word; while( wordListFile &gt;&gt; word ) { if( !word.empty() ) { dict.insert(word); } } wordListFile.close(); } </code></pre> <p>I'm reading in ~200,000 words and this takes about 240 ms on my machine. Is the use of <code>ifstream</code> here efficient? Can I do better? I'm reading about <code>mmap()</code> implementations but I'm not understanding them 100%. The input file is simply text strings with *nix line terminations.</p> <p><strong>EDIT: Follow-up question</strong> for the alternatives being suggested: Would <em>any</em> alternative (minus increasing the stream buffer sizes) imply that I write a parser that examines each character for new-lines? I kind of like the simple syntax of streams, but I can re-write something more nitty-gritty if I have to for speed. Reading the entire file in to memory is a viable option, it's only about 2mb.</p> <p><strong>EDIT #2:</strong> I've found that the slow down for me was due to the set insert, but for those who are still interested in speeding up line by line file IO, please read the answers here AND check out <a href="https://stackoverflow.com/questions/5166263/how-to-get-iostream-to-perform-better">Matthieu M.'s continuation on the topic.</a></p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USJon
    plurals
    1. PL
      singulars
      1. LTLinked
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POHow can I speed up line by line reading of an ASCII file? (C++)
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POHow can I speed up line by line reading of an ASCII file? (C++)
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POHow can I speed up line by line reading of an ASCII file? (C++)
      1. This table or related slice is empty.
      1. VTUpMod
    1. COBiggest gain regarding the reading is to use a large buffer so you don't go through all the O/S layers for every single line (or small buffer) and don't repeatedly pay the performance overhead penalty for that. Dunno what the API for increasing an ifstream's read buffer size is, though, that's why this is a comment, not an answer. But I am sure there is a method to resize the buffer (or assign your own, specifying its size).
      singulars
      1. POHow can I speed up line by line reading of an ASCII file? (C++)
      1. USTheBlastOne
    2. COThe first thing you should do when performance is a problem is to ___profile___. Have a look at where your code spends most of its time. For all we know, it might be in adding the value to your dictionary. Unlikely, I know, but if you need performance, you really need to know.
      singulars
      1. POHow can I speed up line by line reading of an ASCII file? (C++)
      1. USsbi
    3. CO@sbi: It's not that unlikely. In a high-performance statistics app, I've found the Boost version of `unordered_map` to be an order of magnitude slower than Google's `sparsehash` and not much faster than GNU `std::map`.
      singulars
      1. POHow can I speed up line by line reading of an ASCII file? (C++)
      1. USFred Foo
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload