StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POBetter multithreaded use of Python subprocess.Popen & communicate()?
text
Body
copied!<p>I'm running multiple commands which may take some time, in parallel, on a Linux machine running Python 2.6.</p> <p>So, I used <code>subprocess.Popen</code> class and <code>process.communicate()</code> method to parallelize execution of mulitple command groups and capture the output at once after execution.</p> <pre><code>def run_commands(commands, print_lock): # this part runs in parallel. outputs = [] for command in commands: proc = subprocess.Popen(shlex.split(command), stdout=subprocess.PIPE, stderr=subprocess.STDOUT, close_fds=True) output, unused_err = proc.communicate() # buffers the output retcode = proc.poll() # ensures subprocess termination outputs.append(output) with print_lock: # print them at once (synchronized) for output in outputs: for line in output.splitlines(): print(line) </code></pre> <p>At somewhere else it's called like this:</p> <pre><code>processes = [] print_lock = Lock() for ...: commands = ... # a group of commands is generated, which takes some time. processes.append(Thread(target=run_commands, args=(commands, print_lock))) processes[-1].start() for p in processes: p.join() print('done.') </code></pre> <p>The expected result is that each output of a group of commands is displayed at once while execution of them is done in parallel.</p> <p>But from the second output group (of course, the thread that become the second is changed due to scheduling indeterminism), it begins to print without newlines and adding spaces as many as the number of characters printed in each previous line and input echo is turned off -- the terminal state is "garbled" or "crashed". (If I issue <code>reset</code> shell command, it restores normal.)</p> <p>At first, I tried to find the reason from handling of <code>'\r'</code>, but it was not the reason. As you see in my code, I handled it properly using <code>splitlines()</code>, and I confirmed that with <code>repr()</code> function applied to the output.</p> <p>I think the reason is concurrent use of pipes in <code>Popen</code> and <code>communicate()</code> for stdout/stderr. I tried <code>check_output</code> shortcut method in Python 2.7, but no success. Of course, the problem described above does not occur if I serialize all command executions and prints.</p> <p>Is there any better way to handle <code>Popen</code> and <code>communicate()</code> in parallel?</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload