StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>Like @thiton, I don't see the need for a supervisor thread - it seem like an unnecessary complication?</p> <p>This seems to a question of flow-control as well as queue-design. Given what you seem to need, I would go for two producer-consumer queues and a limited number of A_OUTPUT instances. I would create 1000, (say), A_OUTPUT instances and push them onto one P-C queue, forming a thread-safe object pool, at startup, (pool queue). The A threads start, pop an A_OUTPUT from the pool and commence 'fetching raw data from outside world'. When an A thread has got data in its A_OUTPUT, it pushes it onto another P-C queue, (comms queue). A pool of B threads are waiting on the comms queue. When an A_OUTPUT becomes available on the comms queue, a B thread will get it and process it. When the B thread has finished with the data, is pushes the A_OUTPUT back onto the pool queue. The A_OUTPUT instances therefore circulate around, carrying data from one end of the system to the other and then back to the start via the pool queue.</p> <p>Designs like this allow flow-control across multiple threads/pools. There is sufficient 'slack' in the queues to allow bursts of high load, but runaway threads/objects is not possible - if there is too much data flowing through, the A threads will find the pool empty and block on it until A_OUTPUT instances become available - when the are, the A threads will continue to fetch more data.</p> <p>Such a system can be tuned at run time. Adding/removing threads from the A/B thread pools and increasing/decreasing the object pool depth is easy.</p> <p>Oh - an you don't need complex bounded P-C queues. If each queue can hold the number of objects in the pool, that's enough - no more will ever be available.</p> <p>Rgds, Martin</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload