Databases
StackOverflow2013
Postsdbo
POPython popen running out of memory with large output
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POPython popen running out of memory with large output
    primarykey
    data
    text
    <p>I am using the subprocess.popen() function to run an external tool that reads &amp; writes a lot of data (>GB) to stdout. However, I'm finding that the kernel is killing the python process when it runs out of memory:</p> <pre><code>Out of memory: Kill process 8221 (python) score 971 or sacrifice child Killed process 8221 (python) total-vm:8532708kB, anon-rss:3703912kB, file-rss:48kB </code></pre> <p>Since I know I'm handling a large amount of data I've set up popen to write stdout and stderr to files so I'm not using pipes. My code looks something like this:</p> <pre><code>errorFile = open(errorFilePath, "w") outFile = open(outFilePath, "w") #Use Popen to run the command try: procExecCommand = subprocess.Popen(commandToExecute, shell=False, stderr=errorFile, stdout=outFile) exitCode = procExecCommand.wait() except Exception, e: #Write exception to error log errorFile.write(str(e)) errorFile.close() outFile.close() </code></pre> <p>I've tried changing the shell parameter to True and setting the bufsize parameter = -1 also with no luck.</p> <p>I've profiled the memory running this script and via bash and I see big spike in the memory usage when running via Python than compared to bash. </p> <p>I'm not sure what exactly Python is doing to consume so much more memory than the just using bash unless it has something to with trying to write the output to the file? The bash script just pipes the output to a file.</p> <p>I initially found that my swap space was quite low so I increased it and that helped initially, but as the volume of data grows then I start running out of memory again. </p> <p>So is there anything with Python I can do to try and handle these data volumes better, or is it just a case of recommending more memory with plenty of swap space. That or jettison Python altogether.</p> <p>System details:</p> <ul> <li>Ubuntu 12.04</li> <li>Python 2.7.3</li> <li>The tool I'm running is mpileup from samtools.</li> </ul>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. UScraigb
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. COYou could try to run the process as `Popen("myprocess -arg > output",shell=True)`. i.e. send the exact string you would use in bash into `Popen` with `shell=True`.
      singulars
      1. POPython popen running out of memory with large output
      1. USmgilson
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload