Databases
StackOverflow2013
Postsdbo
POParallelization of long running processes and performance optimization
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POParallelization of long running processes and performance optimization
    text
    copied!<p>I would like to parallelize the application that processes multiple video clips frame by frame. Sequence of each frame per clip is important (obviously). I decided to go with TPL Dataflow since I believe this is a good example of dataflow (movie frames being data).</p> <p>So I have one process that loads frames from database (lets say in a batch of 500, all bunched up)</p> <pre><code>Example sequence: |mid:1 fr:1|mid:1 fr:2|mid:2 fr:1|mid:3 fr:1|mid:1 fr:3|mid:2 fr:2|mid:2 fr:3|mid:1 fr:4| </code></pre> <p>and posts them to BufferBlock. To this BufferBlock I have linked ActionBlocks with the filter to have one ActionBlock per MovieID so that I get some kind of data partitioning. Each ActionBlock is sequential, but ideally multiple ActionBlocks for multiple movies can run in parallel.</p> <p>I do have the above described network working and it does run in parallel, but from my calculations only eight to ten ActionBlocks are executing simultaneously. I timed each ActionBlock's running time and its around 100-200ms. What steps can I take to at least double concurrency?</p> <p>I did try converting action delegates to async methods and make database access asynchronous within ActionBlock action delegate but it did not help.</p> <p><strong>EDIT:</strong> I implemented extra level of data partitioning: frames for Movies with Odd IDs are processed on ServerA, frames for Even movies are processed on ServerB. Both instances of the application hit the same database. If my problem was DB IO, then I would not see any improvement in total frames processed count (or very little, under 20%). But I do see it doubling. So this leads me to conclude that Threadpool is not spawning more threads to do more frames in parallel (both servers are quad-cores and profiler shows about 25-30 threads per application).</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload