Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>What you really want is some way to pass exceptions up to the parent process, right? Then you can handle them however you want.</p> <p>If you use <a href="http://docs.python.org/dev/library/concurrent.futures.html"><code>concurrent.futures.ProcessPoolExecutor</code></a>, this is automatic. If you use <a href="http://docs.python.org/dev/library/multiprocessing.html#using-a-pool-of-workers"><code>multiprocessing.Pool</code></a>, it's trivial. If you use explicit <code>Process</code> and <code>Queue</code>, you have to do a bit of work, but it's not <em>that</em> much.</p> <p>For example:</p> <pre><code>def run(self): try: for i in iter(self.inputQueue.get, 'STOP'): # (code that does stuff) 1 / 0 # Dumb error # (more code that does stuff) self.outputQueue.put(result) except Exception as e: self.outputQueue.put(e) </code></pre> <p>Then, your calling code can just read <code>Exception</code>s off the queue like anything else. Instead of this:</p> <pre><code>yield outq.pop() </code></pre> <p>do this:</p> <pre><code>result = outq.pop() if isinstance(result, Exception): raise result yield result </code></pre> <p>(I don't know what your actual parent-process queue-reading code does, because your minimal sample just ignores the queue. But hopefully this explains the idea, even though your real code doesn't actually work like this.)</p> <p>This assumes that you want to abort on any unhandled exception that makes it up to <code>run</code>. If you want to pass back the exception and continue on to the next <code>i in iter</code>, just move the <code>try</code> into the <code>for</code>, instead of around it.</p> <p>This also assumes that <code>Exception</code>s are not valid values. If that's an issue, the simplest solution is to just push <code>(result, exception)</code> tuples:</p> <pre><code>def run(self): try: for i in iter(self.inputQueue.get, 'STOP'): # (code that does stuff) 1 / 0 # Dumb error # (more code that does stuff) self.outputQueue.put((result, None)) except Exception as e: self.outputQueue.put((None, e)) </code></pre> <p>Then, your popping code does this:</p> <pre><code>result, exception = outq.pop() if exception: raise exception yield result </code></pre> <p>You may notice that this is similar to the node.js callback style, where you pass <code>(err, result)</code> to every callback. Yes, it's annoying, and you're going to mess up code in that style. But you're not actually using that anywhere except in the wrapper; all of your "application-level" code that gets values off the queue or gets called inside <code>run</code> just sees normal returns/yields and raised exceptions.</p> <p>You may even want to consider building a <code>Future</code> to the spec of <code>concurrent.futures</code> (or using that class as-is), even though you're doing your job queuing and executing manually. It's not that hard, and it gives you a very nice API, especially for debugging.</p> <p>Finally, it's worth noting that most code built around workers and queues can be made a lot simpler with an executor/pool design, even if you're absolutely sure you only want one worker per queue. Just scrap all the boilerplate, and turn the loop in the <code>Worker.run</code> method into a function (which just <code>return</code>s or <code>raise</code>s as normal, instead of appending to a queue). On the calling side, again scrap all the boilerplate and just <code>submit</code> or <code>map</code> the job function with its parameters.</p> <p>Your whole example can be reduced to:</p> <pre><code>def job(i): # (code that does stuff) 1 / 0 # Dumb error # (more code that does stuff) return result with concurrent.futures.ProcessPoolExecutor(max_workers=1) as executor: results = executor.map(job, range(10)) </code></pre> <p>And it'll automatically handle exceptions properly.</p> <hr> <p>As you mentioned in the comments, the traceback for an exception doesn't trace back into the child process; it only goes as far as the manual <code>raise result</code> call (or, if you're using a pool or executor, the guts of the pool or executor).</p> <p>The reason is that <code>multiprocessing.Queue</code> is built on top of <code>pickle</code>, and pickling exceptions doesn't pickle their tracebacks. And the reason for that is that you can't pickle tracebacks. And the reason for that is that tracebacks are full of references to the local execution context, so making them work in another process would be very hard.</p> <p>So… what can you do about this? Don't go looking for a fully general solution. Instead, think about what you actually need. 90% of the time, what you want is "log the exception, with traceback, and continue" or "print the exception, with traceback, to <code>stderr</code> and <code>exit(1)</code> like the default unhandled-exception handler". For either of those, you don't need to pass an exception at all; just format it on the child side and pass a string over. If you <em>do</em> need something more fancy, work out exactly what you need, and pass just enough information to manually put that together. If you don't know how to format tracebacks and exceptions, see the <a href="http://docs.python.org/2/library/traceback.html"><code>traceback</code></a> module. It's pretty simple. And this means you don't need to get into the pickle machinery at all. (Not that it's very hard to <code>copyreg</code> a pickler or write a holder class with a <code>__reduce__</code> method or anything, but if you don't need to, why learn all that?)</p>
    singulars
    1. This table or related slice is empty.
    1. POPython multiprocessing Process crashes silently
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USabarnert
    1. USabarnert
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POPython multiprocessing Process crashes silently
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    1. COThanks! This is great. But is there any way to print the entire stack trace? It tells me that there's an error now, and what it is, but not WHERE in the Worker class the error occurs.
      singulars
      1. PO
      1. UShendra
    2. CO@npo: I'll add to the answer to explain that.
      singulars
      1. PO
      1. USabarnert
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload