StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>You can go with something like this (just a fast concept, you'll need to add some additional error checking etc.):</p> <pre><code>#include "boost/iostreams/stream.hpp" #include "boost/iostreams/device/mapped_file.hpp" #include "boost/filesystem.hpp" #include "boost/lexical_cast.hpp" double parse_double(const std::string & str) { double value = 0; bool decimal = false; double divisor = 1.0; for (std::string::const_iterator it = str.begin(); it != str.end(); ++it) { switch (*it) { case '.': case ',': decimal = true; break; default: { const int x = *it - '0'; value = value * 10 + x; if (decimal) divisor *= 10; } break; } } return value / divisor; } void process_value(const bool initialized, const std::string & str, std::vector< double > & values) { if (!initialized) { // convert the value count and prepare the output vector const size_t count = boost::lexical_cast< size_t >(str); values.reserve(count); } else { // convert the value //const double value = 0; // ~ 0:20 min const double value = parse_double(str); // ~ 0:35 min //const double value = atof(str.c_str()); // ~ 1:20 min //const double value = boost::lexical_cast< double >(str); // ~ 8:00 min ?!?!? values.push_back(value); } } bool load_file(const std::string & name, std::vector< double > & values) { const int granularity = boost::iostreams::mapped_file_source::alignment(); const boost::uintmax_t chunk_size = ( (256 /* MB */ << 20 ) / granularity ) * granularity; boost::iostreams::mapped_file_params in_params(name); in_params.offset = 0; boost::uintmax_t left = boost::filesystem::file_size(name); std::string value; bool whitespace = true; bool initialized = false; while (left > 0) { in_params.length = static_cast< size_t >(std::min(chunk_size, left)); boost::iostreams::mapped_file_source in(in_params); if (!in.is_open()) return false; const boost::iostreams::mapped_file_source::size_type size = in.size(); const char * data = in.data(); for (boost::iostreams::mapped_file_source::size_type i = 0; i < size; ++i, ++data) { const char c = *data; if (strchr(" \t\n\r", c)) { // c is whitespace if (!whitespace) { whitespace = true; // finished previous value process_value(initialized, value, values); initialized = true; // start a new value value.clear(); } } else { // c is not whitespace whitespace = false; // append the char to the value value += c; } } if (size < chunk_size) break; in_params.offset += chunk_size; left -= chunk_size; } if (!whitespace) { // convert the last value process_value(initialized, value, values); } return true; } </code></pre> <p>Note that your main problem will be the conversion from string to float, which is very slow (insanely slow in the case of boost::lexical_cast). With my custom special parse_double func it is faster, however it only allows a special format (e.g. you'll need to add sign detection if negative values are allowed etc. - or you can just go with atof if all possible formats are needed).</p> <p>If you'll want to parse the file faster, you'll probably need to go for multithreading - for example one thread only parsing the string values and other one or more threads converting the loaded string values to floats. In that case you probably won't even need the memory mapped file, as the regular buffered file read might suffice (the file will be read only once anyway).</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload