StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>fixed my multiprocessing problem - and actually switched the threads. Not sure what actually fixed it thought - i just re-architected everything and made workers and tasks and what not and things are flying now. Here's the basics of what i did:</p> <pre><code>import abc from Queue import Empty, Queue from threading import Thread class AbstractTask(object): """ The base task """ __metaclass__ = abc.ABCMeta @abc.abstractmethod def run_task(self): pass class TaskRunner(object): def __init__(self, queue_size, num_threads=1, stop_on_exception=False): super(TaskRunner, self).__init__() self.queue = Queue(queue_size) self.execute_tasks = True self.stop_on_exception = stop_on_exception # create a worker def _worker(): while self.execute_tasks: # get a task task = None try: task = self.queue.get(False, 1) except Empty: continue # execute the task failed = True try: task.run_task() failed = False finally: if failed and self.stop_on_exception: print('Stopping due to exception') self.execute_tasks = False self.queue.task_done() # start threads for i in range(0, int(num_threads)): t = Thread(target=_worker) t.daemon = True t.start() def add_task(self, task, block=True, timeout=None): """ Adds a task """ if not self.execute_tasks: raise Exception('TaskRunner is not accepting tasks') self.queue.put(task, block, timeout) def wait_for_tasks(self): """ Waits for tasks to complete """ if not self.execute_tasks: raise Exception('TaskRunner is not accepting tasks') self.queue.join() </code></pre> <p>all i do is create a TaskRunner and add tasks to it (thousands of them) and then call wait_for_tasks(). so, obviously in the re-architecture that I did I 'fixed' some other problem that i had. Odd though.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload