Databases
StackOverflow2013
Postsdbo
POPitfalls of trying to use PLINQ over long-running generators?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POPitfalls of trying to use PLINQ over long-running generators?
    primarykey
    data
    text
    <p>I have a few infinite generator methods, including some long-running and infinitely-long-running generators.</p> <pre><code>IEnumerable&lt;T&gt; ExampleOne() { while(true) // this one blocks for a few seconds at a time yield return LongRunningFunction(); } IEnumerable&lt;T&gt; ExampleTwo() { while(true) //this one blocks for a really long time yield return OtherLongRunningFunction(); } </code></pre> <p>My goal is to have an infinite sequence that combines the items from the two examples. Here's what I tried, using PLINQ:</p> <pre><code> IEnumerable&lt;T&gt; combined = new[] { ExampleOne(), ExampleTwo() } .AsParallel() .WithMergeOptions(ParallelMergeOptions.NotBuffered) .WithExecutionMode(ParallelExecutionMode.ForceParallelism) .SelectMany(source =&gt; source.GetRequests()); </code></pre> <p>This seems appropriately combines the two IEnumerables into a new one, with items from <code>IEnumerable</code> #1 and #2 being available whenever they appear in either of the two source <code>IEnumerables</code>:</p> <pre><code>//assuming ExampleTwo yields TWO but happens roughly 5 times //less often then ExampleOne Example output: one one one one one TWO one one one one one one TWO </code></pre> <p>However, it seems like sometimes (usually after many <strong>hours</strong> of running) <code>OtherLongRunningFunction()</code> will go for a long period of time without returning, and under conditions that are hard to reproduce, the <code>combined</code> sequence will block on it rather than continuing to return results from the first <code>LongRunningFunction</code>. It seems that although the combined parallel query started off using two threads, it decided to switch to one thread later on.</p> <p>My first thought was "this is probably a job for RX <code>Observable.Merge</code> and not for PLINQ." But I'd appreciate both answers that show correct alternative ways to handle this situation as well as explanations about the mechanics of how PLINQ can change the degree of parallelism hours after the the start of a query.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USJimmy
    1. USJimmy
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POPitfalls of trying to use PLINQ over long-running generators?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POPitfalls of trying to use PLINQ over long-running generators?
      1. USvoidengine
      1. VTFavorite
    3. VO
      singulars
      1. POPitfalls of trying to use PLINQ over long-running generators?
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI can't say I've got much experience with PLINQ, so this will mostly be unqualified guessing: It appears as if you assume combining `ExampleOne()` and `ExampleTwo()` and running `AsParallel()` on the resulting `IEnumerable` will alternate strictly by returning one result from the 1st and one result from the 2nd. Could it be that this assumption is false? If so, you can end up in a situation where the sequence being processed looks something like _1st 2nd 1st 2nd 1st 1st 1st 2nd 1st 2nd 1st 1st 2nd 1st 2nd 2nd 2nd 2nd..._ And that could explain why it _appears_ as if you're stuck on number two.
      singulars
      1. POPitfalls of trying to use PLINQ over long-running generators?
      1. USNailuj
    2. CO@Nailuj It's actually more like I expect *1 1 1 1 1 1 1 2 1 1 1 1 1 1 1 2* etc, 2s are rather rare, and there can be long periods of times between consecutive 2s. I expect the combined sequence to continue returning 1s, and indeed, this is what actually happens for the most part. But sometimes it stops returning 1s as well.
      singulars
      1. POPitfalls of trying to use PLINQ over long-running generators?
      1. USJimmy
    3. COAnother thought: could it be that PLINQ starts off by processing an equal number of 1s and 2s in parallel, but since the 1s finish much faster, it will appear as the sequence you described. However, whenever a 1 finished, that "spot" will alternating be filled up by a 1 and a 2 respectively. The result will be that in the beginning, it will appear as if you get many 1s and just the rare 2, but in the long run the "queue" of tasks being processed will be filled up with 2s, and thus appear to be stuck? If not exactly like this, could it be somewhat related? Just trying to think out loud :-)
      singulars
      1. POPitfalls of trying to use PLINQ over long-running generators?
      1. USNailuj
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload