Databases
StackOverflow2013
Postsdbo
PODetecting when a child process is waiting for input
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PODetecting when a child process is waiting for input
    primarykey
    data
    text
    <p>I'm writing a Python program for running user-uploaded arbitrary (and thus, at the worst case, unsafe, erroneous and crashing) code on a Linux server. The security questions aside, my objective is to determine, if the code (that might be in any language, compiled or interpreted) writes the correct things to <code>stdout</code>, <code>stderr</code> and other files on given input fed into the program's <code>stdin</code>. After this, I need to display the results to the user.</p> <h1>The current solution</h1> <p>Currently, my solution is to spawn the child process using <code>subprocess.Popen(...)</code> with file handles for the <code>stdout</code>, <code>stderr</code> and <code>stdin</code>. The file behind the <code>stdin</code> handle contains the inputs that the program reads during operation, and after the program has terminated, the <code>stdout</code> and <code>stderr</code> files are read and checked for correctness.</p> <h2>The problem</h2> <p>This approach works otherwise perfectly, but when I display the results, I can't combine the given inputs and outputs so that the inputs would appear in the same places as they would when running the program from a terminal. I.e. for a program like</p> <pre class="lang-py prettyprint-override"><code>print "Hello." name = raw_input("Type your name: ") print "Nice to meet you, %s!" % (name) </code></pre> <p>the contents of the file containing the program's <code>stdout</code> would, after running, be:</p> <pre class="lang-none prettyprint-override"><code>Hello. Type your name: Nice to meet you, Anonymous! </code></pre> <p>given that the contents the file containing the <code>stdin</code> were <code>Anonymous&lt;LF&gt;</code>. So, in short, for the given example code (and, equivalently, for <em>any</em> other code) I want to achieve a result like:</p> <pre class="lang-none prettyprint-override"><code>Hello. Type your name: Anonymous Nice to meet you, Anonymous! </code></pre> <p>Thus, the problem is to detect when the program is waiting for input.</p> <h1>Tried methods</h1> <p>I've tried the following methods for solving the problem:</p> <h2><a href="http://docs.python.org/2/library/subprocess.html#subprocess.Popen.communicate" rel="nofollow noreferrer">Popen.communicate(...)</a></h2> <p>This allows the parent process to separately send data along a <a href="http://docs.python.org/2/library/subprocess.html#subprocess.PIPE" rel="nofollow noreferrer">pipe</a>, but can only be called once, and is therefore not suitable for programs with multiple outputs and inputs - just as can be inferred from the documentation.</p> <h2>Directly reading from <a href="http://docs.python.org/2/library/subprocess.html#subprocess.Popen.stdout" rel="nofollow noreferrer">Popen.stdout</a> and <a href="http://docs.python.org/2/library/subprocess.html#subprocess.Popen.stderr" rel="nofollow noreferrer">Popen.stderr</a> and writing to <a href="http://docs.python.org/2/library/subprocess.html#subprocess.Popen.stdin" rel="nofollow noreferrer">Popen.stdin</a></h2> <p>The documentation warns against this, and the <code>Popen.stdout</code>s <a href="http://docs.python.org/2/library/stdtypes.html#file.read" rel="nofollow noreferrer"><code>.read()</code></a> and <a href="http://docs.python.org/2/library/stdtypes.html#file.readline" rel="nofollow noreferrer"><code>.readline()</code></a> calls seem to block infinitely when the programs starts to wait for input.</p> <h2>Using <a href="http://docs.python.org/2/library/select.html#select.select" rel="nofollow noreferrer"><code>select.select(...)</code></a> to see if the file handles are ready for I/O</h2> <p>This doesn't seem to improve anything. Apparently the pipes are always ready for reading or writing, so <code>select.select(...)</code> doesn't help much here.</p> <h2>Using a different thread for non-blocking reading</h2> <p>As suggested in <a href="https://stackoverflow.com/a/4896288/2096560">this answer</a>, I have tried creating a separate <a href="http://docs.python.org/2/library/threading.html#threading.Thread" rel="nofollow noreferrer">Thread()</a> that stores results from reading from the <code>stdout</code> into a <a href="http://docs.python.org/2/library/queue.html#queue-objects" rel="nofollow noreferrer">Queue()</a>. The output lines before a line demanding user input are displayed nicely, but the line on which the program starts to wait for user input (<code>"Type your name: "</code> in the example above) never gets read.</p> <h2>Using a <a href="http://en.wikipedia.org/wiki/Pseudo_terminal" rel="nofollow noreferrer">PTY</a> slave as the child process' file handles</h2> <p>As directed <a href="https://stackoverflow.com/a/1547764/2096560">here</a>, I've tried <a href="http://docs.python.org/2/library/pty.html#pty.openpty" rel="nofollow noreferrer"><code>pty.openpty()</code></a> to create a pseudo terminal with master and slave file descriptors. After that, I've given the slave file descriptor as an argument for the <code>subprocess.Popen(...)</code> call's <code>stdout</code>, <code>stderr</code> and <code>stdin</code> parameters. Reading through the master file descriptor opened with <a href="http://docs.python.org/2/library/os.html#os.fdopen" rel="nofollow noreferrer"><code>os.fdopen(...)</code></a> yields the same result as using a different thread: the line demanding input doesn't get read.</p> <p><strong>Edit:</strong> Using @Antti Haapala's example of <code>pty.fork()</code> for child process creation instead of <code>subprocess.Popen(...)</code> seems to allow me also read the output created by <code>raw_input(...)</code>.</p> <h2>Using <a href="http://www.noah.org/wiki/pexpect" rel="nofollow noreferrer">pexpect</a></h2> <p>I've also tried the <code>read()</code>, <code>read_nonblocking()</code> and <code>readline()</code> methods (documented <a href="http://pexpect.sourceforge.net/pexpect.html" rel="nofollow noreferrer">here</a>) of a process spawned with pexpect, but the best result, which I got with <code>read_nonblocking()</code>, <s>is the same as before: the line with outputs before wanting the user to enter something doesn't get read.</s> is the same as with a PTY created with <code>pty.fork()</code>: the line demanding input <em>does</em> get read.</p> <p><strong>Edit:</strong> By using <code>sys.stdout.write(...)</code> and <code>sys.stdout.flush()</code> instead of <code>print</code>ing in my <em>master</em> program, which creates the child, seemed to fix the prompt line not getting displayed - it actually got read in both cases, though.</p> <h2>Others</h2> <p>I've also tried <a href="http://docs.python.org/2/library/select.html#select.poll" rel="nofollow noreferrer"><code>select.poll(...)</code></a>, but it seemed that the pipe or PTY master file descriptors are always ready for writing.</p> <h1>Notes</h1> <h2>Other solutions</h2> <ul> <li>What also crossed my mind is to try feeding the input when some time has passed without new output having been generated. This, however, is risky, because there's no way to know if the program is just in the middle of doing a heavy calculation.</li> <li>As @Antti Haapala mentioned in his answer, the <code>read()</code> system call wrapper from glibc could be replaced to communicate the inputs to the master program. However, this doesn't work with statically linked or assembly programs. (Although, now that I think of it, any such calls could be intercepted from the source code and replaced with the patched version of <code>read()</code> - could be painstaking to implement still.)</li> <li>Modifying the Linux kernel code to communicate the <code>read()</code> syscalls to the program is probably insane...</li> </ul> <h2>PTYs</h2> <p>I think the PTY is the way to go, since it fakes a terminal and interactive programs are run on terminals everywhere. The question is, how?</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USmiikkas
    plurals
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. PODetecting when a child process is waiting for input
      1. USfedorqui
      1. VTFavorite
    2. VO
      singulars
      1. PODetecting when a child process is waiting for input
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PODetecting when a child process is waiting for input
      1. USrenqHIT
      1. VTFavorite
    1. CO+1 very well presented question. I do think that PTY is the correct approach because the typed user input appears neither on the stdin or stdout, it is a function of tty/pty echoing. That said, I've not used the `pty` module, so can't give you a proper answer.
      singulars
      1. PODetecting when a child process is waiting for input
      1. USmsw
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload