StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>If you want to run a bunch of jobs using a fixed-size thread pool, you can use <a href="http://docs.python.org/3/library/concurrent.futures.html#threadpoolexecutor" rel="nofollow noreferrer"><code>concurrent.futures.ThreadPoolExecutor</code></a>, like this:</p> <pre><code>from concurrent.futures import ThreadPoolExecutor with ThreadPoolExecutor(max_workers=5) as executor: for work in work_list: executor.submit(work_function, work) </code></pre> <p>If you want to ensure that you make at most one API call a second, then you need to do this from inside your <code>work_function</code>. You can't do it when submitting the job, because you don't know how long the job will queue up waiting for a thread to become available.</p> <p>If it were me, I'd put the rate limiting code into its own class so that it's reusable:</p> <pre><code>from collections import Iterator from threading import Lock import time class RateLimiter(Iterator): """Iterator that yields a value at most once every 'interval' seconds.""" def __init__(self, interval): self.lock = Lock() self.interval = interval self.next_yield = 0 def __next__(self): with self.lock: t = time.monotonic() if t < self.next_yield: time.sleep(self.next_yield - t) t = time.monotonic() self.next_yield = t + self.interval api_rate_limiter = RateLimiter(1) def work_function(work): next(api_rate_limiter) call_api(...) </code></pre> <p><a href="https://docs.python.org/3/library/time.html#time.monotonic" rel="nofollow noreferrer"><code>time.monotonic</code></a> was introduced in Python 3.3; in older versions of Python you could use <a href="https://docs.python.org/3/library/time.html#time.time" rel="nofollow noreferrer"><code>time.time</code></a> but this can jump backwards when the system clock changes, so you would need to ensure that this doesn't cause overlong sleeps:</p> <pre><code> time.sleep(min(self.next_yield - t, self.interval)) </code></pre>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload