StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POPython subprocess.Popen "OSError: [Errno 12] Cannot allocate memory"
text
Body
copied!<p><strong>Note:</strong> This question was originally asked <a href="https://stackoverflow.com/questions/1216794/python-subprocess-popen-erroring-with-oserror-errno-12-cannot-allocate-memory">here</a> but the bounty time expired even though an acceptable answer was not actually found. I am re-asking this question including all details provided in the original question.</p> <p>A python script is running a set of class functions every 60 seconds using the <a href="http://docs.python.org/library/sched.html" rel="noreferrer">sched</a> module:</p> <pre><code># sc is a sched.scheduler instance sc.enter(60, 1, self.doChecks, (sc, False)) </code></pre> <p>The script is running as a daemonised process using the code <a href="http://www.jejik.com/articles/2007/02/a_simple_unix_linux_daemon_in_python/" rel="noreferrer">here</a>.</p> <p>A number of class methods that are called as part of doChecks use the <a href="http://docs.python.org/library/subprocess.html" rel="noreferrer">subprocess</a> module to call system functions in order to get system statistics:</p> <pre><code>ps = subprocess.Popen(['ps', 'aux'], stdout=subprocess.PIPE).communicate()[0] </code></pre> <p>This runs fine for a period of time before the entire script crashing with the following error:</p> <pre><code>File "/home/admin/sd-agent/checks.py", line 436, in getProcesses File "/usr/lib/python2.4/subprocess.py", line 533, in __init__ File "/usr/lib/python2.4/subprocess.py", line 835, in _get_handles OSError: [Errno 12] Cannot allocate memory </code></pre> <p>The output of free -m on the server once the script has crashed is:</p> <pre><code>$ free -m total used free shared buffers cached Mem: 894 345 549 0 0 0 -/+ buffers/cache: 345 549 Swap: 0 0 0 </code></pre> <p>The server is running CentOS 5.3. I am unable to reproduce on my own CentOS boxes nor with any other user reporting the same problem.</p> <p>I have tried a number of things to debug this as suggested in the original question:</p> <ol> <li><p>Logging the output of free -m before and after the Popen call. There is no significant change in memory usage i.e. memory is not gradually being used up as the script runs.</p></li> <li><p>I added close_fds=True to the Popen call but this made no difference - the script still crashed with the same error. Suggested <a href="https://stackoverflow.com/questions/1216794/python-subprocess-popen-erroring-with-oserror-errno-12-cannot-allocate-memory/1297785#1297785">here</a> and <a href="https://stackoverflow.com/questions/1216794/python-subprocess-popen-erroring-with-oserror-errno-12-cannot-allocate-memory/1278961#1278961">here</a>.</p></li> <li><p>I checked the rlimits which showed (-1, -1) on both RLIMIT_DATA and RLIMIT_AS as suggested <a href="https://stackoverflow.com/questions/1216794/python-subprocess-popen-erroring-with-oserror-errno-12-cannot-allocate-memory/1270442#1270442">here</a>.</p></li> <li><p><a href="http://www.zenoss.com/community/wiki/common-error-messages-and-solutions/oserror-errno-12-cannot-allocate-memory-in-popen2.py/" rel="noreferrer">An article</a> suggested the having no swap space might be the cause but swap is actually available on demand (according to the web host) and this was also suggested as a bogus cause <a href="https://stackoverflow.com/questions/1216794/python-subprocess-popen-erroring-with-oserror-errno-12-cannot-allocate-memory/1270171#1270171">here</a>.</p></li> <li><p>The processes are being closed because that is the behaviour of using .communicate() as backed up by the Python source code and comments <a href="https://stackoverflow.com/questions/1216794/python-subprocess-popen-erroring-with-oserror-errno-12-cannot-allocate-memory/1216824#1216824">here</a>.</p></li> </ol> <p>The entire checks can be found at on <a href="http://github.com/dmytton/sd-agent/blob/82f5ff9203e54d2adeee8cfed704d09e3f00e8eb/checks.py" rel="noreferrer">GitHub here</a> with the getProcesses function defined from line 442. This is called by doChecks() starting at line 520.</p> <p>The script was run with strace with the following output before the crash:</p> <pre><code>recv(4, "Total Accesses: 516662\nTotal kBy"..., 234, 0) = 234 gettimeofday({1250893252, 887805}, NULL) = 0 write(3, "2009-08-21 17:20:52,887 - checks"..., 91) = 91 gettimeofday({1250893252, 888362}, NULL) = 0 write(3, "2009-08-21 17:20:52,888 - checks"..., 74) = 74 gettimeofday({1250893252, 888897}, NULL) = 0 write(3, "2009-08-21 17:20:52,888 - checks"..., 67) = 67 gettimeofday({1250893252, 889184}, NULL) = 0 write(3, "2009-08-21 17:20:52,889 - checks"..., 81) = 81 close(4) = 0 gettimeofday({1250893252, 889591}, NULL) = 0 write(3, "2009-08-21 17:20:52,889 - checks"..., 63) = 63 pipe([4, 5]) = 0 pipe([6, 7]) = 0 fcntl64(7, F_GETFD) = 0 fcntl64(7, F_SETFD, FD_CLOEXEC) = 0 clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb7f12708) = -1 ENOMEM (Cannot allocate memory) write(2, "Traceback (most recent call last"..., 35) = 35 open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory) open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory) open("/usr/lib/python24.zip/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/usr/lib/python2.4/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/usr/lib/python2.4/plat-linux2/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory) open("/usr/lib/python2.4/lib-tk/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/usr/lib/python2.4/lib-dynload/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/usr/lib/python2.4/site-packages/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) write(2, " File \"/usr/bin/sd-agent/agent."..., 52) = 52 open("/home/admin/sd-agent/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory) open("/usr/bin/sd-agent/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory) open("/usr/lib/python24.zip/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/usr/lib/python2.4/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/usr/lib/python2.4/plat-linux2/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory) open("/usr/lib/python2.4/lib-tk/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/usr/lib/python2.4/lib-dynload/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/usr/lib/python2.4/site-packages/daemon.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) write(2, " File \"/home/admin/sd-agent/dae"..., 60) = 60 open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory) open("/usr/bin/sd-agent/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory) open("/usr/lib/python24.zip/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/usr/lib/python2.4/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/usr/lib/python2.4/plat-linux2/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory) open("/usr/lib/python2.4/lib-tk/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/usr/lib/python2.4/lib-dynload/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/usr/lib/python2.4/site-packages/agent.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) write(2, " File \"/usr/bin/sd-agent/agent."..., 54) = 54 open("/usr/lib/python2.4/sched.py", O_RDONLY|O_LARGEFILE) = 8 write(2, " File \"/usr/lib/python2.4/sched"..., 55) = 55 fstat64(8, {st_mode=S_IFREG|0644, st_size=4054, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000 read(8, "\"\"\"A generally useful event sche"..., 4096) = 4054 write(2, " ", 4) = 4 write(2, "void = action(*argument)\n", 25) = 25 close(8) = 0 munmap(0xb7d28000, 4096) = 0 open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory) open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory) open("/usr/lib/python24.zip/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/usr/lib/python2.4/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/usr/lib/python2.4/plat-linux2/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory) open("/usr/lib/python2.4/lib-tk/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/usr/lib/python2.4/lib-dynload/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/usr/lib/python2.4/site-packages/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) write(2, " File \"/usr/bin/sd-agent/checks"..., 60) = 60 open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory) open("/usr/bin/sd-agent/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory) open("/usr/lib/python24.zip/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/usr/lib/python2.4/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/usr/lib/python2.4/plat-linux2/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOMEM (Cannot allocate memory) open("/usr/lib/python2.4/lib-tk/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/usr/lib/python2.4/lib-dynload/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) open("/usr/lib/python2.4/site-packages/checks.py", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or directory) write(2, " File \"/usr/bin/sd-agent/checks"..., 64) = 64 open("/usr/lib/python2.4/subprocess.py", O_RDONLY|O_LARGEFILE) = 8 write(2, " File \"/usr/lib/python2.4/subpr"..., 65) = 65 fstat64(8, {st_mode=S_IFREG|0644, st_size=39931, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000 read(8, "# subprocess - Subprocesses with"..., 4096) = 4096 read(8, "lso, the newlines attribute of t"..., 4096) = 4096 read(8, "code < 0:\n print >>sys.st"..., 4096) = 4096 read(8, "alse does not exist on 2.2.0\ntry"..., 4096) = 4096 read(8, " p2cread\n # c2pread <-"..., 4096) = 4096 write(2, " ", 4) = 4 write(2, "errread, errwrite)\n", 19) = 19 close(8) = 0 munmap(0xb7d28000, 4096) = 0 open("/usr/lib/python2.4/subprocess.py", O_RDONLY|O_LARGEFILE) = 8 write(2, " File \"/usr/lib/python2.4/subpr"..., 71) = 71 fstat64(8, {st_mode=S_IFREG|0644, st_size=39931, ...}) = 0 mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb7d28000 read(8, "# subprocess - Subprocesses with"..., 4096) = 4096 read(8, "lso, the newlines attribute of t"..., 4096) = 4096 read(8, "code < 0:\n print >>sys.st"..., 4096) = 4096 read(8, "alse does not exist on 2.2.0\ntry"..., 4096) = 4096 read(8, " p2cread\n # c2pread <-"..., 4096) = 4096 read(8, "table(self, handle):\n "..., 4096) = 4096 read(8, "rrno using _sys_errlist (or siml"..., 4096) = 4096 read(8, " p2cwrite = None, None\n "..., 4096) = 4096 write(2, " ", 4) = 4 write(2, "self.pid = os.fork()\n", 21) = 21 close(8) = 0 munmap(0xb7d28000, 4096) = 0 write(2, "OSError", 7) = 7 write(2, ": ", 2) = 2 write(2, "[Errno 12] Cannot allocate memor"..., 33) = 33 write(2, "\n", 1) = 1 unlink("/var/run/sd-agent.pid") = 0 close(3) = 0 munmap(0xb7e0d000, 4096) = 0 rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x589978}, {0xb89a60, [], SA_RESTORER, 0x589978}, 8) = 0 brk(0xa022000) = 0xa022000 exit_group(1) = ? </code></pre>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload