Databases
StackOverflow2013
Postsdbo
POSlow processing a for loop that utilizes findstr
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POSlow processing a for loop that utilizes findstr
    primarykey
    data
    text
    <p>I've got a somewhat weird case, where a for-loop is incredibly slow when I use findstr as the string for DO.</p> <p>Its worth mentioning that the file (<code>old-file.xml</code>) that I'm processing contains about 200 000 lines.</p> <p>This part is blazing fast, but can be rendered slower if I remove <code>| find /c ":"</code></p> <pre><code>rem find total number of lines in xml-file findstr /n ^^ old-file.xml | find /c ":" &gt; "temp-count.txt" set /p lines=&lt; "temp-count.txt" </code></pre> <p>The code which is slow looks like this and I can't use the pipe trick above. It seems like the slow part is the <code>for</code> itself, as i'm not seeing any progress in the title bar until after 10 min.</p> <pre><code>setlocal DisableDelayedExpansion rem start replacing wrong dates with correct date for /f "usebackq Tokens=1* Delims=:" %%i in (`"findstr /n ^^ old-file.xml"`) do ( rem cache the value of each line in a variable set read-line=%%j set line=%%i rem restore delayed expansion setlocal EnableDelayedExpansion rem write progress in title bar title Processing line: !line!/%lines% rem remove trailing line number rem set read-line=!read-line:*:=! for /f "usebackq" %%i in ("%tmpfile%") do ( rem replace all wrong dates with correct dates set read-line=!read-line:%%i=%correctdate%! ) rem write results to new file echo(!read-line!&gt;&gt;"Updated-file.xml" rem end local endlocal ) </code></pre> <p><strong>EDIT:</strong></p> <p>Further investigation showed me that using this single line that should display the current line number being looped takes about 10 minutes on my 8MB file of 200 000 lines. That's just for getting it to start displaying the lines.</p> <pre><code>for /f "usebackq Tokens=1* Delims=:" %%i in (`"findstr /n ^^ old-file.xml"`) do echo %%i </code></pre> <p>So it seems like <code>findstr</code> is writing screen output hidden for the user, but visible for the <code>for</code>-loop. How can I prevent that from happening while still getting the same results?</p> <p><strong>EDIT 2: Solution</strong></p> <p>The solution as proposed by <a href="https://stackoverflow.com/a/8300722/487650">Aacini</a> and finally revised by me.</p> <p>This is a snippet from a much bigger script. Wrong dates are retrieved in another loop. And total number of lines are also retrieved from another loop.</p> <pre><code>setlocal enabledelayedexpansion rem this part is for snippet only, dates are generated from another loop in final script echo 2069-04-29 &gt; dates-tmp.txt echo 2069-04-30 &gt;&gt; dates-tmp.txt findstr /n ^^ Super-Large-File.xml &gt; out.tmp set tmpfile=dates-tmp.txt set correctdate=2011-11-25 set wrong-dates= rem hardcoded total number of lines set lines=186442 for /F %%i in (%tmpfile%) do ( set wrong-dates=!wrong-dates! %%i ) rem process each line in out.tmp and loop them through :ProcessLines call :ProcessLines &lt; out.tmp rem when finished with above call for each line in out.tmp, goto exit goto ProcessLinesEnd :ProcessLines for /L %%l in (1,1,%lines%) do ( set /P read-line= rem write progress in title bar title Processing line: %%l/%lines% for %%i in (%wrong-dates%) do ( rem replace all wrong dates with correct dates set read-line=!read-line:%%i=%correctdate%! ) rem write results to new file echo(!read-line:*:=!&gt;&gt;"out2.tmp" ) rem end here and continue below goto :eof :ProcessLinesEnd echo this should not be printed until call has ended :exit exit /b </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USNiklasJ
    plurals
    1. This table or related slice is empty.
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POSlow processing a for loop that utilizes findstr
      1. This table or related slice is empty.
      1. VTUpMod
    1. COWhat filesize has your `old-file.xml`? You could try to add an `echo I'm alive` into your loop, to see if the `findstr` is the problem
      singulars
      1. POSlow processing a for loop that utilizes findstr
      1. USjeb
    2. CO`old-file.xml` contains 200 000 lines, and is about 8 MB in size. I'm having `title` inside the for-loop instead of `echo I'm Alive`
      singulars
      1. POSlow processing a for loop that utilizes findstr
      1. USNiklasJ
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload