Databases
StackOverflow2013
Postsdbo
POParallel.Invoke - Dynamically creating more 'threads'
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POParallel.Invoke - Dynamically creating more 'threads'
    text
    copied!<p>I am educating myself on Parallel.Invoke, and parallel processing in general, for use in current project. I need a push in the right direction to understand how you can dynamically\intelligently allocate more parallel 'threads' as required.</p> <p>As an example. Say you are parsing large log files. This involves reading from file, some sort of parsing of the returned lines and finally writing to a database.</p> <p>So to me this is a typical problem that can benefit from parallel processing.</p> <p>As a simple first pass the following code implements this.</p> <pre><code>Parallel.Invoke( ()=&gt; readFileLinesToBuffer(), ()=&gt; parseFileLinesFromBuffer(), ()=&gt; updateResultsToDatabase() ); </code></pre> <p>Behind the scenes </p> <ol> <li>readFileLinesToBuffer() reads each line and stores to a buffer.</li> <li>parseFileLinesFromBuffer comes along and consumes lines from buffer and then let's say it put them on another buffer so that updateResultsToDatabase() can come along and consume this buffer.</li> </ol> <p>So the code shown assumes that each of the three steps uses the same amount of time\resources but lets say the parseFileLinesFromBuffer() is a long running process so instead of running just one of these methods you want to run two in parallel.</p> <p>How can you have the code intelligently decide to do this based on any bottlenecks it might perceive?</p> <p>Conceptually I can see how some approach of monitoring the buffer sizes might work, spawning a new 'thread' to consume the buffer at an increased rate for example...but I figure this type of issue has been considered in putting together the TPL library.</p> <p>Some sample code would be great but I really just need a clue as to what concepts I should investigate next. It looks like maybe the System.Threading.Tasks.TaskScheduler holds the key?</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload