StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p><strong>Reconsidered answer</strong></p> <p>I've been rethinking my original answer below. I still suspect that using fewer threads would probably be a good idea, but as you're just <em>moving</em> files, it shouldn't actually be that IO intensive. It's possible that just <em>listing</em> the files is taking a lot of disk work.</p> <p>However, I doubt that you're really running out of memory for the files. How much memory have you got? How much memory is the process taking up? How many threads are you using, and how many cores do you have? (Using significantly more threads than you have cores is a bad idea, IMO.)</p> <p>I suggest the following plan of attack:</p> <ul> <li>Work out where the bottlenecks actually are. Try fetching the list of files but not doing the moving them. See how hard the disk is hit, and how long it takes.</li> <li>Experiment with different numbers of threads, with a queue of directories still to process.</li> <li>Keep an eye on the memory use and garbage collections. The Windows performance counters for the CLR are good for this.</li> </ul> <p><strong>Original answer</strong></p> <p>Rewriting in C or C++ wouldn't help. Using multiple processes wouldn't help. What you're doing is akin to giving a single processor a hundred threads - except you're doing it with the disk instead.</p> <p>It makes sense to parallelise tasks which use IO if there's <em>also</em> a fair amount of computation involved, but if it's already disk bound, asking the disk to work with lots of files at the same time is only going to make things worse.</p> <p>You may be interested in a benchmark (<a href="http://msmvps.com/blogs/jon_skeet/archive/2009/03/20/benchmarking-io-buffering-vs-streaming.aspx" rel="nofollow noreferrer">description</a> and <a href="http://msmvps.com/blogs/jon_skeet/archive/2009/03/24/buffering-vs-streaming-benchmark-first-results.aspx" rel="nofollow noreferrer">initial results</a>) I've recently been running, testing "encryption" of individual lines of a file. When the level of "encryption" is low (i.e. it's hardly doing any CPU work) the best results are always with a single thread.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload