Databases
StackOverflow2013
Postsdbo
POParsing text from CMemFile line by line
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POParsing text from CMemFile line by line
    primarykey
    data
    text
    <p>I have got a huge text file loaded into a <code>CMemFile</code> object and would like to parse it line by line (separated by newline chars).<br> Originally it is a zip file on disk, and I unzip it into memory to parse it, therefore the CMemFile.</p> <p>One working way to read line by line is this (m_file is a smart pointer to a <code>CMemFile</code>):</p> <pre><code> CArchive archive(m_file.get(), CArchive::load); CString line; while(archive.ReadString(line)) { ProcessLine(string(line)); } </code></pre> <p>Since it takes a lot of time, I tried to write my own routine:</p> <pre><code> const UINT READSIZE = 1024; const char NEWLINE = '\n'; char readBuffer[READSIZE]; UINT bytesRead = 0; char *posNewline = NULL; const char* itEnd = readBuffer + READSIZE; ULONGLONG currentPosition = 0; ULONGLONG newlinePositionInBuffer = 0; do { currentPosition = m_file-&gt;GetPosition(); bytesRead = m_file-&gt;Read(&amp;readBuffer, READSIZE); if(bytesRead == 0) break; // EOF posNewline = std::find(readBuffer, readBuffer + bytesRead, NEWLINE); if(posNewline != itEnd) { // found newline ProcessLine(string(readBuffer, posNewline)); newlinePositionInBuffer = posNewline - readBuffer + 1; // +1 to skip \r m_file-&gt;Seek(currentPosition + newlinePositionInBuffer, CFile::begin); } } while(true); </code></pre> <p>Measuring the performance showed both methods take about the same time...</p> <p><strong>Can you think of any performance improvements or a faster way to do the parsing?</strong></p> <p>Thanks for any advice</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USnabulke
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POParsing text from CMemFile line by line
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POParsing text from CMemFile line by line
      1. USjcibar
      1. VTFavorite
    1. COHow big is "huge"? How long are the lines? How long is "a lot of time"? I mean, maybe you are already doing it as fast as you can. Also, I imagine the "speed test" is done in release mode, isn't it? And finally, what does `ProcessLine()` do? Maybe there are lots of memory allocations or something else that's taking too much time.
      singulars
      1. POParsing text from CMemFile line by line
      1. USMikMik
    2. CO@MikMik: Yes, it is difficult to judge whether my solution is already good or not. To answer a few of your questions: uncompressed files are about 100 MB, line length upto 250 chars, tested in release mode and optimized for speed. ProcessLine is an empty function. Right now it takes about 1 second/file to parse.
      singulars
      1. POParsing text from CMemFile line by line
      1. USnabulke
    3. COOften there are completely different solutions to reach your goal. Thats why I'm asking.
      singulars
      1. POParsing text from CMemFile line by line
      1. USnabulke
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload