Databases
StackOverflow2013
Postsdbo
POPython process abruptly killed during execution
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POPython process abruptly killed during execution
    primarykey
    data
    text
    <p>I'm new to python and am facing what seems to be a memory leakage error. I've written a simple script that is trying to fetch multiple columns from a postgres database and then proceeds to perform simple subtraction on these columns and store the result in a temporary variable which is being written to a file. I need to do this on multiple pairs of columns from the db and I'm using a list of lists to store the different column names. </p> <p>I'm loop over the individual elements of this list until the list is exhausted. While I'm getting valid results(by valid I mean that the output file contains the expected values) for the first few column pairs, the program abruptly gets "Killed" somewhere in between execution. Code below:</p> <pre><code>varList = [ ['table1', 'col1', 'col2'], ['table1', 'col3', 'col4'], ['table2', 'col1', 'col2'], # .. # and many more such lines # .. ['table2', 'col3', 'col4']] try: conn = psycopg2.connect(database='somename', user='someuser', password='somepasswd') c = conn.cursor() for listVar in varList: c.execute("SELECT %s FROM %s" %(listVar[1], listVar[0])) rowsList1 = c.fetchall(); c.execute("SELECT %s FROM %s" %(listVar[2], listVar[0])) rowsList2 = c.fetchall(); outfile = file('%s__%s' %(listVar[1], listVar[2]), 'w') for i in range(0, len(rowsList1)): if rowsList1[i][0] == None or rowsList2[i][0] == None: timeDiff = -1 else: timestamp1 = time.mktime(rowsList1[i][0].timetuple()) timestamp2 = time.mktime(rowsList2[i][0].timetuple()) timeDiff = timestamp2 - timestamp1 outfile.write(str(timeDiff) + '\n') outfile.close(); del rowsList1, rowsList2 #numpy.savetxt('output.dat', column_stack(rows)) except psycopg2.DatabaseError, e: print 'Error %s' % e sys.exit(1) finally: if conn: conn.close() </code></pre> <p>My initial guess was that there was some form of memory leak and in an attempt to fix this, I added a del statement on the two large arrays hoping that the memory gets properly collected. This time, I got slightly better outputs(by slightly better I mean that more output files were created for the db column pairs). However, after the 10th or 11th pair of columns, my program was "Killed" again. Can someone tell me what could be wrong here. Is there a better way of getting this done? Any help is appreciated.</p> <p>PS: I know that this is a fairly inefficient implementation as I'm looping many times, but I needed something quick and dirty for proof of concept. </p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USAnilRamakrishna
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POPython process abruptly killed during execution
      1. This table or related slice is empty.
      1. VTUpMod
    1. COHow much data do you expect to process here?
      singulars
      1. POPython process abruptly killed during execution
      1. USGreg Hewgill
    2. COcursor.fetchall() is probably the killer as it is going to try to duplicate your entire DB in core. fetchone() is probably what you want since you are processing it a row at a time anyway.
      singulars
      1. POPython process abruptly killed during execution
      1. USmsw
    3. COIs there any reason you can't just get the database to calculate your time differences in SQL? Then you could just iterate over the cursor and write out the values to the output file.
      singulars
      1. POPython process abruptly killed during execution
      1. USJames Henstridge
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload