Databases
StackOverflow2013
Postsdbo
POWhy am I leaking memory with this python loop?
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POWhy am I leaking memory with this python loop?
    text
    copied!<p>I am writing a custom file system crawler, which gets passed millions of globs to process through sys.stdin. I'm finding that when running the script, its memory usage increases massively over time and the whole thing crawls practically to a halt. I've written a minimal case below which shows the problem. Am I doing something wrong, or have I found a bug in Python / the glob module? (I am using python 2.5.2).</p> <pre><code> #!/usr/bin/env python import glob import sys import gc previous_num_objects = 0 for count, line in enumerate(sys.stdin): glob_result = glob.glob(line.rstrip('\n')) current_num_objects = len(gc.get_objects()) new_objects = current_num_objects - previous_num_objects print "(%d) This: %d, New: %d, Garbage: %d, Collection Counts: %s"\ % (count, current_num_objects, new_objects, len(gc.garbage), gc.get_count()) previous_num_objects = current_num_objects </code></pre> <p>The output looks like:</p> <pre> (0) This: 4042, New: 4042, Python Garbage: 0, Python Collection Counts: (660, 5, 0) (1) This: 4061, New: 19, Python Garbage: 0, Python Collection Counts: (90, 6, 0) (2) This: 4064, New: 3, Python Garbage: 0, Python Collection Counts: (127, 6, 0) (3) This: 4067, New: 3, Python Garbage: 0, Python Collection Counts: (130, 6, 0) (4) This: 4070, New: 3, Python Garbage: 0, Python Collection Counts: (133, 6, 0) (5) This: 4073, New: 3, Python Garbage: 0, Python Collection Counts: (136, 6, 0) (6) This: 4076, New: 3, Python Garbage: 0, Python Collection Counts: (139, 6, 0) (7) This: 4079, New: 3, Python Garbage: 0, Python Collection Counts: (142, 6, 0) (8) This: 4082, New: 3, Python Garbage: 0, Python Collection Counts: (145, 6, 0) (9) This: 4085, New: 3, Python Garbage: 0, Python Collection Counts: (148, 6, 0) </pre> <p>Every 100th iteration, 100 objects are freed, so <code>len(gc.get_objects()</code> increases by 200 every 100 iterations. <code>len(gc.garbage)</code> never changes from 0. The 2nd generation collection count increases slowly, while the 0th and 1st counts go up and down.</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload