Databases
StackOverflow2013
Postsdbo
POQuickly loading numerical data in C++
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POQuickly loading numerical data in C++
    text
    copied!<p>I'm working on a feature detection program that uses statistical models of what various sorts of landmarks in images look like. The model uses around 100 different landmarks, and the relevant data for each landmark consists of 16 matrices of doubles, each of size about 160x160.</p> <p>I'm currently using one text file for each landmark, and storing the values from each matrix as a space-separated line. To read the data, I read a line at a time from each file and pass it to a function that generates a stringstream from the line and then reads the values of the matrix from this stream one at a time. </p> <p>On my computer this takes about 90 seconds to load the ~40 million doubles the model uses. There must surely be a far quicker way of doing this, but I've not found anything useful from googling, and I've got no experience with this sort of thing. </p> <p>I'd be very grateful for any suggestions.</p> <p><strong>Edit:</strong> Loki asked me to post code, so I've shown it below. loadFromFile is called once for each landmark. The first line of each landmark file states how many levels the model uses for this landmark (each level uses four matrices; there are four levels by default). It's a horrible mess, but I'm not sure why this is so spectularly slow.</p> <pre><code>void loadFromFile(string filename) { ifstream modelData(filename.c_str(), ifstream::in); string line; getline(modelData,line); int numberOfLevels = atoi(line.c_str()); for(size_t ii = 0; ii &lt; numberOfLevels; ++ii) readProfileStats(modelData); modelData.close(); } void readProfileStats(ifstream&amp; fileStream) { string line; getline(fileStream, line); Vector meanProfile = readMatrixFromString(line); getline(fileStream, line); Matrix principalComponents = readMatrixFromString(line); getline(fileStream, line); Matrix covarianceMatrixInverse = readMatrixFromString(line); m_statsLevels.push_back(ProfileStats(meanProfile, principalComponents, covarianceMatrixInverse)); } Matrix readMatrixFromString(const string&amp; line) { stringstream stream(line); size_t numRows; size_t numCols; stream &gt;&gt; numRows; stream &gt;&gt; numCols; Matrix matrix(numRows,numCols); for(int ii = 0; ii &lt; numRows; ++ii) { for(int jj = 0; jj &lt; numCols; ++jj) stream &gt;&gt; matrix(ii,jj); } return matrix; } </code></pre>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload