Databases
StackOverflow2013
Postsdbo
POdesign issues in producer-consumer problem
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POdesign issues in producer-consumer problem
    primarykey
    data
    text
    <p>I am working on a little multi-threading project.The system could be divided into 2 sub parts, A and B. the data flows from A to B.<br> A part keeps fetching raw data from outside world, do some transformation and and then generate thousands of new data, let's call it A_OUTPUT.<br> B part do some calculations based on each A_OUTPUT and then generate even more data, may be ten times the number of A_OUTPUT. </p> <p>I am confused about how to synchronize the 2 parts.<br> My own design is to create a work queue as well as a lock protecting the queue between the two sub parts. Also create a event to indicate whether the work queue is empty or not. </p> <p>A part consists multiple threads, each thread fetch data from outside and generate A_OUTPUT, each time a single A thread generate a A_OUTPUT, the thread obtain the queue lock, push the A_OUTPUT into the queue, release the lock, and then trigger the event. </p> <p>B part consists a supervisor thread and several worker threads, the supervisor thread was first blocked on the event. after the event was triggered, the supervisor thread lock the queue, fetch all A_OUTPUTS of the queue, release the lock, dispatch A_OUTPUTS to worker threads, and then wait on the event again. </p> <p>The problem of this design is obvious, the supervisor thread of B will be racing with multiple threads of A to win the queue lock. maybe when B finally own the lock, there was already ten or more A_OUTPUTs in the queue, and the most aged A_OUTPUT was generated a long time ago. I want each A_OUTPUT to be processed as fast as possible. </p> <p>I know I could divide the work queue into several smaller queues or add more B supervisor threads into the lock battle to shorten the average time each A_OUTPUT wait before it get processed. but might there exist a more appropriate design? </p> <p>And another question, does there exists any paradigm or design pattern for different purpose multi-thread programs?</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USJason
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POdesign issues in producer-consumer problem
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POdesign issues in producer-consumer problem
      1. This table or related slice is empty.
      1. VTUpMod
    1. COThere are 2 concepts involved here: queue locking and synchronizing. I am not proficient in C++, but java deals with these two concepts in the implementation of this queue object, called 'Blocking Queue': http://download.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/BlockingQueue.html
      singulars
      1. POdesign issues in producer-consumer problem
      1. USDanC
    2. COHow do you intend to control the number of B worker threads created, couldn't that just spiral out of control. "fetch all A_OUTPUTS of the queue" doesn't sound like a good idea if processing of A_OUTPUTS is going to take time, since we are looking at unbounded thread creation. I think we need some kind of a threadpool here.
      singulars
      1. POdesign issues in producer-consumer problem
      1. USAbhijeet Kashnia
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload