Databases
StackOverflow2013
Postsdbo
POEfficient approach to multithreaded set difference
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POEfficient approach to multithreaded set difference
    primarykey
    data
    text
    <p>I have a finite set of consumer threads each consuming a job. Once they process the job, they have a list of subjobs that were listed in the consumed job. I need to add the subjobs from that list that I don't already have in the database. There are 3 million in the database, so getting the list of which ones aren't already in the database is slow. I don't mind each thread blocking on that call, but since I have a race condition (see code) I have to lock them all on that slow call, so they can only call that section one at a time and my program crawls. What can I do to fix this so the threads don't slow down for that call? I tried a queue, but since the threads are pushing out lists of jobs faster than the computer can determine which ones should be added to the database, I end up with a queue that keeps growing and never empties.</p> <p>My code:</p> <pre><code>IEnumerable&lt;string&gt; getUniqueJobNames(IEnumerable&lt;job&gt; subJobs, int setID) { return subJobs.Select(el =&gt; el.name) .Except(db.jobs.Where(el =&gt; el.set_ID==setID).Select(el =&gt; el.name)); } //...consumer thread i lock(lockObj) { var uniqueJobNames = getUniqueJobNames(consumedJob.subJobs, consumerSetID); //if there was a context switch here to some thread i+1 // and that thread found uniqueJobs that also were found in thread i // then there will be multiple copies of the same job added in the database. // So I put this section in a lock to prevent that. saveJobsToDatabase(uniqueJobName, consumerSetID); } //continue consumer thread i... </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USbrandon
    1. USbrandon
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POEfficient approach to multithreaded set difference
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POEfficient approach to multithreaded set difference
      1. USEugene S.
      1. VTFavorite
    3. VO
      singulars
      1. POEfficient approach to multithreaded set difference
      1. This table or related slice is empty.
      1. VTUpMod
    1. COit's not clear to me what you are trying todo, can you explain again WHAT you are trying todo, but without the info on HOW you currently do it, just so the actual tasks becomes more clear
      singulars
      1. POEfficient approach to multithreaded set difference
      1. USntziolis
    2. COCan't you first get the list of existing jobs, then compile a list of "new" subjobs in parallel and, finally, save the new jobs?
      singulars
      1. POEfficient approach to multithreaded set difference
      1. USGert Arnold
    3. COThe problem is I don't know which ones are new unless I compare them against the database using except. I could compile a list of all subjobs that come up but when I finally want to compare that list against the database, it wouldn't finish by the time the next list came around. They're building up faster than I can run the Except method regardless if I cache the list for later or run it immediately. Actually if I run it immediately, the consumers would run faster and the issue is compounded. I'm guessing there's some datastructure that could help, or just a different algorithm.
      singulars
      1. POEfficient approach to multithreaded set difference
      1. USbrandon
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload