StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POPitfalls of trying to use PLINQ over long-running generators?
text
Body
copied!<p>I have a few infinite generator methods, including some long-running and infinitely-long-running generators.</p> <pre><code>IEnumerable<T> ExampleOne() { while(true) // this one blocks for a few seconds at a time yield return LongRunningFunction(); } IEnumerable<T> ExampleTwo() { while(true) //this one blocks for a really long time yield return OtherLongRunningFunction(); } </code></pre> <p>My goal is to have an infinite sequence that combines the items from the two examples. Here's what I tried, using PLINQ:</p> <pre><code> IEnumerable<T> combined = new[] { ExampleOne(), ExampleTwo() } .AsParallel() .WithMergeOptions(ParallelMergeOptions.NotBuffered) .WithExecutionMode(ParallelExecutionMode.ForceParallelism) .SelectMany(source => source.GetRequests()); </code></pre> <p>This seems appropriately combines the two IEnumerables into a new one, with items from <code>IEnumerable</code> #1 and #2 being available whenever they appear in either of the two source <code>IEnumerables</code>:</p> <pre><code>//assuming ExampleTwo yields TWO but happens roughly 5 times //less often then ExampleOne Example output: one one one one one TWO one one one one one one TWO </code></pre> <p>However, it seems like sometimes (usually after many <strong>hours</strong> of running) <code>OtherLongRunningFunction()</code> will go for a long period of time without returning, and under conditions that are hard to reproduce, the <code>combined</code> sequence will block on it rather than continuing to return results from the first <code>LongRunningFunction</code>. It seems that although the combined parallel query started off using two threads, it decided to switch to one thread later on.</p> <p>My first thought was "this is probably a job for RX <code>Observable.Merge</code> and not for PLINQ." But I'd appreciate both answers that show correct alternative ways to handle this situation as well as explanations about the mechanics of how PLINQ can change the degree of parallelism hours after the the start of a query.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload