Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>Some assumptions:</p> <ul> <li><p>From your example data, you are receiving movie frames (and possibly the frames in the movies) out of order</p></li> <li><p>Your <a href="http://msdn.microsoft.com/en-us/library/hh194684.aspx" rel="nofollow"><code>ActionBlock&lt;T&gt;</code></a> instances are generic; they all call the same method for processing, you just create a list of them based on each movie id (you have a list of movie ids beforehand) like so:</p></li> </ul> <pre><code>// The movie IDs IEnumerable&lt;int&gt; movieIds = ...; // The actions. var actions = movieIds.Select( i =&gt; new { Id = i, Action = new ActionBlock&lt;Frame&gt;(MethodToProcessFrame) }); // The buffer block. BufferBlock&lt;Frame&gt; buffer = ...; // Link everything up. foreach (var action in actions) { // Not necessary in C# 5.0, but still, good practice. // The copy of the action. var actionCopy = action; // Link. bufferBlock.LinkTo(actionCopy.Action, f =&gt; f.MovieId == actionCopy.Id); } </code></pre> <p>If this is the case, you're creating too many <code>ActionBlock&lt;T&gt;</code> instances which aren't being given work; because your frames (and possibly movies) are out-of-order, you aren't guaranteed that all of the <code>ActionBlock&lt;T&gt;</code> instances will have work to do.</p> <p>Additionally, when you create an <code>ActionBlock&lt;T&gt;</code> instance it's going to be created with a <a href="http://msdn.microsoft.com/en-us/library/system.threading.tasks.dataflow.executiondataflowblockoptions.maxdegreeofparallelism.aspx" rel="nofollow"><code>MaxDegreeOfParallelism</code></a> of 1, meaning that it's thread safe because only one thread can access the block at the same time.</p> <p>Additionally, the TPL DataFlow library ultimately relies on the <a href="http://msdn.microsoft.com/en-us/library/dd321424.aspx" rel="nofollow"><code>Task&lt;TResult&gt;</code> class</a>, which schedules by default on the thread pool. The thread pool is going to do a few things here:</p> <ul> <li><p>Make sure that all processor cores are saturated. This is <em>very</em> different from making sure that your <code>ActionBlock&lt;T&gt;</code> instances are saturated and <em>this</em> is the metric you should be concerned with</p></li> <li><p>Make sure that while the processor cores are saturated, make sure that the work is distributed evenly, as well as make sure that not <em>too</em> many concurrent tasks are executing (context switches are expensive).</p></li> </ul> <p>It also looks like your method that processes your movies is generic, and it doesn't matter what frame from what movie is passed in (if it <em>does</em> matter, then you need to update your question with that, as it changes a lot of things). This would also mean that it's thread-safe.</p> <p>Also, if it can be assumed that the processing of one frame doesn't rely on the processing of any previous frames (or, it looks like the frames of the movie come in order) you can use a <em>single</em> <code>ActionBlock&lt;T&gt;</code> but tweak up the <code>MaxDegreeOfParallelism</code> value, like so:</p> <pre><code>// The buffer block. BufferBlock&lt;Frame&gt; buffer = ...; // Have *one* ActionBlock&lt;T&gt; var action = new ActionBlock&lt;Frame&gt;(MethodToProcessFrame, // This is where you tweak the concurrency: new ExecutionDataflowBlockOptions { MaxDegreeOfParallelism = 4, } ); // Link. No filter needed. bufferBlock.LinkTo(action); </code></pre> <p>Now, your <code>ActionBlock&lt;T&gt;</code> will <em>always</em> be saturated. Granted, any responsible task scheduler (the thread pool by default) is still going to limit the maximum amount of concurrency, but it's going to do as much as it can reasonably do at the same time.</p> <p>To that end, if your action is <em>truly</em> thread safe, you can set the <code>MaxDegreeOfParallelism</code> to <a href="http://msdn.microsoft.com/en-us/library/system.threading.tasks.dataflow.dataflowblockoptions.unbounded.aspx" rel="nofollow"><code>DataflowBlockOptions.Unbounded</code></a>, like so:</p> <pre><code>// Have *one* ActionBlock&lt;T&gt; var action = new ActionBlock&lt;Frame&gt;(MethodToProcessFrame, // This is where you tweak the concurrency: new ExecutionDataflowBlockOptions { // We're thread-safe, let the scheduler determine // how nuts we can go. MaxDegreeOfParallelism = DataflowBlockOptions.Unbounded, } ); </code></pre> <p>Of course, all of this assumes that everything else is optimal (I/O reads/writes, etc.)</p>
    singulars
    1. This table or related slice is empty.
    1. POParallelization of long running processes and performance optimization
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USsvick
    1. UScasperOne
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POParallelization of long running processes and performance optimization
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI do state in my question that "Sequence of each frame per clip is important". That means that I cannot process frame 2 of movie 1 before frame 1 of movie 1. And I am relying on the output of frame 1 processing to process frame 2 of the same movie. So order in which movies are processed doesn't matter, but order of frames for any given movie matters. That's why I'm partitioning data using ActionBlocks and Filtered Links
      singulars
      1. PO
      1. USDimitri
    2. COOn another note, using my current architecture, I commented out data access code, doubling productivity. So that leads me to believe its a DB IO issue
      singulars
      1. PO
      1. USDimitri
    3. CO@Dimitri If it *is* a DB IO issue, then that makes this question "Too Localized" and it should be closed as such. I missed the part about sequences (and it's not obvious, as not all video processing is dependent on the last clip, depending on what you're trying to do, as the output can be reordered), I'll rework the answer to account for that but if you enter it into the `BufferBlock<T>` in order, then you will get it fed in that order. You just have to make sure you're not processing frames concurrently from the same movie. Also, this will limit just how concurrent you can get, obviously.
      singulars
      1. PO
      1. UScasperOne
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload