Databases
StackOverflow2013
Postsdbo
POReducing system time usage of Python Subprocess
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POReducing system time usage of Python Subprocess
    primarykey
    data
    text
    <p>I have a python script that uses multiprocessing's pool.map( ... ) to run a large number of calculations in parallel. Each of these calculations consists of the python script setting up input for a fortran program, using subprocess.popen( ... , stdin=PIPE, stdout=PIPE, stderr=PIPE ) to run the program, dump the input to it and read the output. Then the script parses the output, gets the needed numbers, then does it again for the next run.</p> <pre><code>def main(): #Read a configuration file #do initial setup pool = multiprocessing.Pool(processes=maxProc) runner = CalcRunner( things that are the same for each run ) runNumsAndChis = pool.map( runner, xrange(startRunNum, endRunNum)) #dump the data that makes it past a cut to disk class CalcRunner(object): def __init__(self, stuff): #setup member variables def __call__(self, runNumber): #get parameters for this run params = self.getParams(runNum) inFileLines = [] #write the lines of the new input file to a list makeInputFile(inFileLines, ... ) process = subprocess.Popen(cmdString, bufsize=0, stdin=subprocess.PIPE, ... ) output = proc.communicate( "".join(inFileLines) ) #get the needed numbers from stdout chi2 = getChiSq(output[0]) return [runNumber, chi2] ... </code></pre> <p>Anyways, on to the reason for the question. I submit this script to a grid engine system to break this huge parameter space sweep into 1000, 12 core (I choose 12 since most of the grid is 12 cores), tasks. When a single task runs on a single 12 core machine about 1/3 of the machine's time is spent doing system stuff, and the other 2/3 of the time is doing the user calculations, presumably setting up inputs to ECIS (the aforementioned FORTRAN code), running ECIS, and parsing the output of ECIS. However, sometimes 5 tasks get sent to a 64 core machine to utilize 60 of its cores. On that machine 40% of the time is spent doing system stuff and 1-2% doing user stuff.</p> <p>First of all, where are all the system calls coming from? I tried writing a version of the program that runs ECIS once per separate thread and keeps piping new input to it and it spends FAR more time in system (and is slower overall), so it doesn't seem like it is due to all the process creation and deletion. Second of all, how do I go about decreasing the amount of time spent on system calls?</p> <p>At a guess, the open a process once and keep sending input to it was slower because I had to turn gfortran's output buffering off to get anything from the process, nothing else worked (short of modifying the fortran code... which isn't happening).</p> <p>The OS on my home test machines where I developed this is Fedora 14. The OS on the grid machines is a recent version of Red Hat.</p> <p>I have tried playing around with bufsize, setting it to -1 (system defaults), 0 (unbuffered), 1 (line by line), and 64Kb that does not seem to change things.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USJames Matta
    1. USJames Matta
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. COIf you truly want performance you will have to look at low level languages like C, not python to remove the overheads. As general as it may seem, that is the reality of it.
      singulars
      1. POReducing system time usage of Python Subprocess
      1. USTymoteusz Paul
    2. COYou should say which OS you're using, so people can suggest appropriate tools to investigate where the system time is going.
      singulars
      1. POReducing system time usage of Python Subprocess
      1. USTim Peters
    3. CO@Puciek, if the system time is going to I/O (whether files or pipes), C isn't going to help. After all, CPython's I/O is already done **by** C code ;-)
      singulars
      1. POReducing system time usage of Python Subprocess
      1. USTim Peters
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload