Databases
StackOverflow2013
Postsdbo
POLooping through File1.txt and File2.txt is really slow. Both files are 280MB
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POLooping through File1.txt and File2.txt is really slow. Both files are 280MB
    text
    copied!<p>I have 2 large text files with 400,000 lines of text in each file. In File2.txt, I need to find the line containing the userId from the current line in File1.txt. Once I've located the correct line in File2.txt, I do some calculations and write the line to a new text file.</p> <p>The code I've created to do this runs really slow. I've tried rewriting it various ways but it always chugs along and never finishes. How can I do this fast?</p> <pre><code>private void btnExecute_Click(object sender, EventArgs e) { string line1 = ""; string line2 = ""; //the new text file we are creating. Located in IVR_Text_Update\bin\Debug StreamWriter sw = new StreamWriter("NewFile.txt"); //the new text file which contains the registrants which need removing StreamWriter sw_removeRegs = new StreamWriter("RemoveRegistrants.txt"); //address has changed so we write the line to the address file StreamWriter sw_addressChange = new StreamWriter("AddressChanged.txt"); List&lt;string&gt; lines_secondFile = new List&lt;string&gt;(); using (StreamReader sr = new StreamReader(openFileDialog2.FileName)) { string line; while ((line = sr.ReadLine()) != null) { lines_secondFile.Add(line); } } //loop through the frozen file one line at a time while ((line1 = sr1.ReadLine()) != null) { //get the line from the update file, assign it to line2 //function accepts (userId, List) line2 = getLine(line1.Substring(3, 8), lines_secondFile); //if line2 is null then userId was not found therefore we write //the line to Remove Registrants file if (line2 == null) { sw_removeRegs.Write(line1 + Environment.NewLine); } //address between the two lines was found to be different so we still write //them to the new text file but don't update codes else if (line1.Substring(93, 53) != line2.Substring(93, 53)) { sw_addressChange.Write(line1 + Environment.NewLine); sw.Write(line1 + Environment.NewLine); } //test for null then write the new line in our new text file else if ((line1 != null) &amp;&amp; (line2 != null)) { sw.Write(line1.Substring(0, 608) + line2.Substring(608, 9) + line2.Substring(617, 9) + line2.Substring(626, 9) + line2.Substring(635, 9) + line2.Substring(644, 9) + line2.Substring(653, 9) + line2.Substring(662, 9) + line2.Substring(671, 9) + line2.Substring(680, 9) + line1.Substring(680, 19) + Environment.NewLine); } } textBox1.Text = "Finished."; sr1.Close(); sw.Close(); sw_removeRegs.Close(); sw_addressChange.Close(); } //returns the line from the update file which has the corresponding userId //from the frozen file string getLine(string userId, List&lt;string&gt; lines_secondFile) { foreach (string currentLine in lines_secondFile) { if (currentLine.Contains(userId)) { return currentLine; } } return null; } </code></pre>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload