StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>Concerning the mechanics of PLINQ:</p> <p>I encounter the same problem: I have a sequence whose items require uneven processing time, some of which longer by orders of magnitude. I experience thread starvation, much more reproducible on an 8-core processor than on 4-core, although it may happen on a 4-core as well after many hours of processing. Some threads might pick up work again after a while. Note that dynamic chunking is used, as in the example. </p> <p>Observation: Starvation more probably happens at completion of consecutive very long-running work items.</p> <p>The MSDN topic <a href="http://msdn.microsoft.com/en-us/library/ff963552.aspx" rel="nofollow noreferrer">Parallel Loops</a> sheds some light:</p> <p><em>Be careful if you use parallel loops with individual steps that take several seconds or more. This can occur with I/O-bound workloads as well as lengthy calculations. If the loops take a long time, you may experience an unbounded growth of worker threads due to a heuristic for preventing thread starvation that's used by the .NET ThreadPool class's thread injection logic. This heuristic steadily increases the number of worker threads when work items of the current pool run for long periods of time. The motivation is to add more threads in cases where everything in the thread pool is blocked. Unfortunately, if work is actually proceeding, more threads may not necessarily be what you want. The .NET Framework can't distinguish between these two situations.</em></p> <p>I still don't know the details, but i think that the underlying ThreadPool's heuristics don't reason well for very long running work items, failing to accommodate threads for next iterations due to some upper limit not properly adapted, thus having iterations queued. I don't have Visual Studio access to the 8-core machine where the problem is reproducible easier. I have not yet been able to reproduce the problem under Visual Studio debugging on a 4-core machine. Investigation continues.</p> <p>For more details, the <a href="https://stackoverflow.com/questions/3488381/does-the-task-parallel-library-or-plinq-take-other-processes-into-account">"Does the Task Parallel Library (or PLINQ) take other processes into account?"</a> topic is highly relevant.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload