StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>So, while what Rick has suggested is definitely an important point, another thing I think is missing is the discussion of <a href="http://blogs.msdn.com/b/pfxteam/archive/2007/12/02/6558579.aspx" rel="noreferrer">partitioning</a>.</p> <p><code>Parallel::ForEach</code> will use a default <a href="http://msdn.microsoft.com/en-us/library/system.collections.concurrent.partitioner.aspx" rel="noreferrer"><code>Partitioner<T></code></a> implementation which, for an <code>IEnumerable<T></code> which has no known length, will use a chunk partitioning strategy. What this means is each worker thread which <code>Parallel::ForEach</code> is going to use to work on the data set will read some number of elements from the <code>IEnumerable<T></code> which will then only be processed by that thread (ignoring work stealing for now). It does this to save the expense of constantly having to go back to the source and allocate some new work and schedule it for another worker thread. So, usually, this is a good thing.However, in your specific scenario, imagine you're on a quad core and you've set <a href="http://msdn.microsoft.com/en-us/library/system.threading.tasks.paralleloptions.maxdegreeofparallelism.aspx" rel="noreferrer"><code>MaxDegreeOfParallelism</code></a> to 4 threads for your work and now each of those pulls a chunk of 100 elements from your <code>IEnumerable<T></code>. Well, that's 100-400 megs right there just for that particular worker thread, right?</p> <p>So how do you solve this? Easy, you <a href="http://msdn.microsoft.com/en-us/library/dd997411.aspx" rel="noreferrer">write a custom <code>Partitioner<T></code> implementation</a>. Now, chunking is still useful in your case, so you probably don't want to go with a single element partitioning strategy because then you would introduce overhead with all the task coordination necessary for that. Instead I would write a configurable version that you can tune via an appsetting until you find the optimal balance for your workload. The good news is that, while writing such an implementation is pretty straightfoward, you don't actually have to even write it yourself because the PFX team already did it and <a href="http://code.msdn.microsoft.com/ParExtSamples" rel="noreferrer">put it into the parallel programming samples project</a>.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload