Databases
StackOverflow2013
Postsdbo
POWhy do my subclassed multiprocessing Processes hang?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POWhy do my subclassed multiprocessing Processes hang?
    primarykey
    data
    text
    <p>I have written a Python program that employs multiprocessing via subclassed Process objects. I am currently trying to improve the error handling, but it is causing me no end of frustration for reasons I cannot comprehend.</p> <p>Bear with me while I lay out some sample code:</p> <p>My subclassed Process worker:</p> <pre><code># worker.py from multiprocessing import Process, Queue import sys import traceback # other imports as necessary class Worker(Process): def __init__(self, inputQueue, outputQueue): try: super(Worker, self).__init__() self.inputQueue = inputQueue self.outputQueue = outputQueue #etc 1/0 # dumb error that causes the process to crash #etc except Exception as e: exc_type, exc_value, exc_traceback = sys.exc_info() e = traceback.format_exception(exc_type, exc_value, exc_traceback, limit = 2) # Forward the error message to the Engine class self.outputQueue.put({ 'type' : 'error', 'data' : e }) def run(self): try: for i in iter(self.inputQueue.get, 'STOP'): # Do stuff self.outputQueue.put({ 'type' : 'result', 'data' : stuff }) except Exception as e: exc_type, exc_value, exc_traceback = sys.exc_info() e = traceback.format_exception(exc_type, exc_value, exc_traceback, limit = 2) # Forward the error message to the Engine class self.outputQueue.put({ 'type' : 'error', 'data' : e }) </code></pre> <p>This is my Engine class that is the parent of the Worker objects:</p> <pre><code># engine.py # imports as necessary class Engine: def __init__(self) # Other arguments as necessary # Initialise the class self.processors = [] def run(self): try: # Fill the inputQueue with a number of jobs # and initialise the output queue # etc # Start the Worker processes for i in range(numberOfProcesses): p = worker.Worker(inputQueue, outputQueue) self.processors.append(p) p.start() inputQueue.put('STOP') # Process outputQueue numberOfJobs = 6 # (say) while numberOfJobs: result = outputQueue.get() if result['type'] == 'error': print result raise Exception elif result['type'] == 'result': # Process result appropriately numberOfJobs -= 1 for p in self.processors: p.join() except Exception as e: raise </code></pre> <p>Configuration file that runs the whole thing:</p> <pre><code># configuration.py if __name__ == "__main__": # Initialise other parameters as necessary try: # Initialise an instance of engine.Engine eng = engine.Engine(arguments as necessary) eng.run() except Exception as e: print e print 'Finished UNSUCCESSFULLY.' else: print 'Finished SUCCESSFULLY.' </code></pre> <p>For brevity, I left a few imports and other things out of the example code. The error handling machinery in worker.py is derived from my earlier question <a href="https://stackoverflow.com/questions/15536295/python-multiprocessing-process-crashes-silently">here</a>.</p> <p>When I run the program, the worker processes execute and when one hits the 1/0 error in <code>Worker.__init__</code> (or in <code>Worker.run()</code>, if there was one in there), it grabs the traceback and puts it into the output queue just fine. <code>Engine.run()</code> prints the error message as it's supposed to. The problem is that is where it all ends. Engine <em>should</em> raise a <em>new</em>, generic, error and (I think) pass it up to the code in configuration.py, whereupon the program should exit via the except statement in that file (printing a message that says 'Finished UNSUCCESSFULLY').</p> <p>Instead, what <em>actually</em> happens is that <code>Engine.run()</code> prints the error message derived from the Worker class, then the program simply hangs and CPU usage drops to 0%. Nothing else happens. I recognise that this is most likely a problem with the other Worker processes not quitting (or whatever they ought to do).</p> <p>What mystifies me is that if I don't have the <code>except Exception as e</code> block in the Engine class, and if I don't have the <code>try-except-else</code> blocks in the configuration.py file, <code>Engine.run()</code> prints the traceback from the Worker object and then crashes properly (doesn't hang) with the generic <code>raise Exception</code> statement.</p> <p>Essentially what I'd like to do is get <code>Engine.run()</code> to pass the generic raise Exception up to the configuration file so that the program exits more gracefully.</p> <p>Someone help me!</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. UShendra
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. COso `p.join()` hangs, right?
      singulars
      1. POWhy do my subclassed multiprocessing Processes hang?
      1. USUser
    2. COWell, I guess so.
      singulars
      1. POWhy do my subclassed multiprocessing Processes hang?
      1. UShendra
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload