Databases
StackOverflow2013
Postsdbo
POFile read over network very slow
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POFile read over network very slow
    primarykey
    data
    text
    <p>Im having an interesting problem with reading a large file (~400 mb) that's on a network drive. Orginally, I fed the full network address into a FileInfo and used the CopyTo function to transfer it to a local temp drive and then read it. This seems to work okay, its not slow but its not fast - just meh. The CopyTo function would get the computer running the program's network utilization consistantly up above 50%, which is pretty good.</p> <p>In order to speed up the process I tried to read the network file directly into a Memory Stream to cut out the middle man so to speak. When I tried this (using the asynchronous copy pattern described <a href="https://stackoverflow.com/a/1247042/643619">here</a>), it is hilariously slow. My network utilization never even tops 2% - its almost like something is throttling me. FYI, I watched my network utilization when directly copying the same file via windows explorer and it hit like 80-90%... not sure what's happening here. Below is the asynchronous copy code I used:</p> <pre><code>string line; List&lt;string&gt; results = new List&lt;string&gt;(); Parser parser = new Parser(QuerySettings.SelectedFilters, QuerySettings.SearchTerms, QuerySettings.ExcludedTerms, QuerySettings.HighlightedTerms); byte[] ActiveBuffer = new byte[60 * 1024]; byte[] BackBuffer = new byte[60 * 1024]; byte[] WriteBuffer = new byte[60 * 1024]; MemoryStream memStream = new MemoryStream(); FileStream fileStream = new FileStream(fullPath, FileMode.Open, FileSystemRights.Read, FileShare.None, 60 * 1024, FileOptions.SequentialScan); int Readed = 0; IAsyncResult ReadResult; IAsyncResult WriteResult; ReadResult = fileStream.BeginRead(ActiveBuffer, 0, ActiveBuffer.Length, null, null); do { Readed = fileStream.EndRead(ReadResult); WriteResult = memStream.BeginWrite(ActiveBuffer, 0, Readed, null, null); WriteBuffer = ActiveBuffer; if (Readed &gt; 0) { ReadResult = fileStream.BeginRead(BackBuffer, 0, BackBuffer.Length, null, null); BackBuffer = Interlocked.Exchange(ref ActiveBuffer, BackBuffer); } memStream.EndWrite(WriteResult); } while (Readed &gt; 0); StreamReader streamReader = new StreamReader(memStream); while ((line = streamReader.ReadLine()) != null) { if (parser.ParseResults(line)) results.Add(line); } fileStream.Flush(); fileStream.Close(); memStream.Flush(); memStream.Close(); return results; </code></pre> <p><strong>UPDATE</strong> As per the comments I just tried the following. It only had my network utilization at about 10-15%... why so low?</p> <pre><code>MemoryStream memStream = new MemoryStream(); FileStream fileStream = File.OpenRead(fullPath); fileStream.CopyTo(memStream); memStream.Seek(0, 0); StreamReader streamReader = new StreamReader(memStream); Parser parser = new Parser(QuerySettings.SelectedFilters, QuerySettings.SearchTerms, QuerySettings.ExcludedTerms, QuerySettings.HighlightedTerms); while ((line = streamReader.ReadLine()) != null) { if (parser.ParseResults(line)) results.Add(line); } </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USHershizer33
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POFile read over network very slow
      1. This table or related slice is empty.
      1. VTApproveEditSuggestion
    2. VO
      singulars
      1. POFile read over network very slow
      1. This table or related slice is empty.
      1. VTUpMod
    1. COGiven that writing to a `MemoryStream` will be very fast indeed, what happens if you just try reading *synchronously*?
      singulars
      1. POFile read over network very slow
      1. USJon Skeet
    2. CONot seeing the point of the async stuff, considering *you're waiting in the same thread for it to finish* when you call `EndRead` or `EndWrite` outside of the callback. That's kinda the very definition of synchronous.
      singulars
      1. POFile read over network very slow
      1. UScHao
    3. CO@JonSkeet Okay I tried something different as you suggested (i think). When you get a chance please read the update to the original post
      singulars
      1. POFile read over network very slow
      1. USHershizer33
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload