Databases
StackOverflow2013
Postsdbo
POWhat's the pythonic way to deal with worker processes that must coordinate their tasks?
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POWhat's the pythonic way to deal with worker processes that must coordinate their tasks?
    text
    copied!<p>I'm currently learning Python (from a Java background), and I have a question about something I would have used threads for in Java.</p> <p>My program will use workers to read from some web-service some data periodically. Each worker will call on the web-service at various times periodically.</p> <p>From what I have read, it's preferable to use the <code>multiprocessing</code> module and set up the workers as independent processes that get on with their data-gathering tasks. On Java I would have done something conceptually similar, but using threads. While it appears I can use threads in Python, I'll lose out on multi-cpu utilisation.</p> <p>Here's the guts of my question: The web-service is throttled, viz., the workers must not call on it more than x times per second. What is the best way for the workers to check on whether they may request data?</p> <p>I'm confused as to whether this should be achieved using: </p> <ul> <li>Pipes as a way to communicate to some other 'managing object', which monitors the total calls per second.</li> <li>Something along the lines of <code>nmap</code>, to share some data/value between the processes that describes if they may call the web-service.</li> <li>A <code>Manager()</code> object that monitors the calls per seconds and informs workers if they have permission to make their calls.</li> </ul> <p>Of course, I guess this may come down to how I keep track of the calls per second. I suppose one option would be for the workers to call a function on some other object, which makes the call to the web-service and records the current number of calls/sec. Another option would be for the function that calls the web-service to live within each worker, and for them to message a managing object every time they make a call to the web-service.</p> <p>Thoughts welcome! </p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload