Databases
StackOverflow2013
Postsdbo
POFast string splitting with multiple delimiters
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POFast string splitting with multiple delimiters
    primarykey
    data
    text
    <p>I investigated some time here on StackOverflow to find good algorithms to split strings with multiple delimiters into a <code>vector&lt; string &gt;</code>. I also found some methods:</p> <p>The Boost way:</p> <pre><code>boost::split(vector, string, boost::is_any_of(" \t")); </code></pre> <p>the <code>getline</code> method:</p> <pre><code>std::stringstream ss(string); std::string item; while(std::getline(ss, item, ' ')) { vector.push_back(item); } </code></pre> <p>the tokenize way of Boost:</p> <pre><code>char_separator&lt;char&gt; sep(" \t"); tokenizer&lt;char_separator&lt;char&gt;&gt; tokens(string, sep); BOOST_FOREACH(string t, tokens) { vector.push_back(t); } </code></pre> <p>and the cool STL way:</p> <pre><code> istringstream iss(string); copy(istream_iterator&lt;string&gt;(iss), istream_iterator&lt;string&gt;(), back_inserter&lt;vector&lt;string&gt; &gt;(vector)); </code></pre> <p>and the method of Shadow2531 (see the linked topic).</p> <p>Most of them came from <a href="https://stackoverflow.com/questions/236129/how-to-split-a-string">this topic</a>. But they unfortunately don't solve my problem:</p> <ul> <li><p>Boost's split is easy to use but with the big data (about 1.5*10^6 single elements in best cases) and about 10 delimiters I am using it's horrific slow.</p></li> <li><p>The <code>getline</code>, STL and Shadow2531's method have the problem that I can only use one single char as delimiter. I need a few more.</p></li> <li><p>Boost's tokenize is even more horrific in the aspect of speed. It took 11 seconds with 10 delimiters to split a string into 1.5*10^6 elements.</p></li> </ul> <p>So I don't know what to do: I want to have a really fast string splitting algorithm with multiple delimiters. </p> <p>Is Boost's split the maximum or is there a way to do it <strong>faster</strong>?</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USPaul
    plurals
    1. PL
      singulars
      1. LTLinked
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    3. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POFast string splitting with multiple delimiters
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POFast string splitting with multiple delimiters
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POFast string splitting with multiple delimiters
      1. This table or related slice is empty.
      1. VTUpMod
    1. COHave you tried multi threading? For instance split your delim into N groups, and for each group run a thread to split the string by that delim group and populate a list, then recombine the lists afterwards?
      singulars
      1. POFast string splitting with multiple delimiters
      1. USNick Banks
    2. COThere must be some faster way. This: `cat loremipsum_big.txt | ruby -e "ary = Array.new; ARGF.each {|x| ary << x.split(/a| /)}; puts ary" | wc -l` creates 2.516.715 elements, pushes them into an array in 3.74 seconds in a Virtualboxed Ubuntu.
      singulars
      1. POFast string splitting with multiple delimiters
      1. USkaratedog
    3. COHow about some of the examples from the following article: http://www.codeproject.com/KB/recipes/Tokenizer.aspx They are very efficient. The String Toolkit Library makes complex string processing in C++ simple and easy.
      singulars
      1. POFast string splitting with multiple delimiters
      1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload