StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POCan I split/merge output streams of subprocess.Popen?
text
Body
copied!<p>I'm writing a wrapper class for use with a workflow manager. I would like to log output from an application (child process executed via <code>subprocess.Popen</code>) in a certain way:</p> <ul> <li><code>stdout</code> of the child should go to a log file and to <code>stdout</code> of the parent,</li> <li><code>stderr</code> of the child should go to a different logfile, but also to <code>stdout</code> of the parent.</li> </ul> <p>I.e. all output from the child should end up merged on <code>stdout</code> (like with <code>subprocess.Popen(..., stderr=subprocess.STDOUT)</code>, so I can reserve <code>stderr</code> for log messages from the wrapper itself. On the other hand, the child's streams should go to different files to allow separate validation.</p> <p>I've tried using a "Tee" helper class to tie two streams (<code>stdout</code> and the log file) together, so that <code>Tee.write</code> writes to both streams. However, this cannot be passed to <code>Popen</code> because "subprocess" uses OS-level functions for writing (see here: <a href="http://bugs.python.org/issue1631" rel="nofollow noreferrer">http://bugs.python.org/issue1631</a>).</p> <p>The problem with my current solution (code snippet below, adapted mostly from <a href="http://alexandre.deverteuil.net/blogue/archives/2011/04/03/monitor_python_subprocess_output_streams_in_real-time/index.html" rel="nofollow noreferrer">here</a>) is that output on <code>stdout</code> may not appear in the right order.</p> <p>How can I overcome this? Or should I use an altogether different approach? (If I stick with the code below, how do I choose a value for the number of bytes in <code>os.read</code>?)</p> <pre><code>import subprocess, select, sys, os call = ... # set this process = subprocess.Popen(call, stdout=subprocess.PIPE, stderr=subprocess.PIPE) logs = {process.stdout: open("out.log", "w"), process.stderr: open("err.log", "w")} done = {process.stdout: False, process.stderr: False} while (process.poll() is None) or (not all(done.values())): ready = select.select([process.stdout, process.stderr], [], [])[0] for stream in ready: data = os.read(stream.fileno(), 1) if data: sys.stdout.write(data) logs[stream].write(data) else: done[stream] = True logs[process.stdout].close() logs[process.stderr].close() </code></pre> <p>By the way, <a href="https://stackoverflow.com/a/7730201/1149840">this solution</a> using "fcntl" has not worked for me. And I couldn't quite figure out how to adapt <a href="https://stackoverflow.com/a/4896288/1149840">this solution</a> to my case yet, so I haven't tried it.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload