StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POPython memory leak when converting and saving numpy array as image
text
Body
copied!<p>I have a simple setup:</p> <p>Producer reads a set of images from a camera, and concatenates them using numpy.row_stack(). The produced array is put into Queue.Queue.</p> <p>Consumer constantly waits for data using Queue.get() and as soon as it receives data it saves the images to a file. For whatever reason the process leaks memory like there is no tomorrow. <strike>If I don't put images into the thread queue no leaking occurs. Apparently it is NOT the queue that is leaking, but something in my save image function. Specifically it is in the creating the temporary file part.</strike> Here is the implementation:</p> <pre><code>def _createTmpFile(self, image): filename = '/tmp/imgTemp.gray' with open(filename, 'wb') as fp: scaled = image * 16 big = numpy.asmatrix(scaled, dtype='>i2') fp.write(big.tostring()) return filename def _saveImage(self, filename, image): tmpFile = self._createTmpFile(image) try: # Call a blocking sub process command to convert the image: command = ('convert -size %dx%d -depth 16 -endian MSB' % ( image.shape[1], image.shape[0])).split(' ') command += [tmpFile, '/tmp/' + os.path.basename(filename)] log.debug("command being called: %s" % str(command)) subprocess.call(command) os.unlink(tmpFile) log.info("Image %s is saved." % filename) except Exception as e: log.exception() log.warning("Failed to save image %s" % filename) </code></pre> <p>I don't see anything particular standing out to me, which would cause a memory leak. Any ideas?</p> <p>UPDATE: After a thorough analysis the problem turned out be pretty simple - producer was creating images at a much faster rate than consumer could save them, thus filling up the queue. When I would cancel the request for images I would drain the queue, but it seems like the GC was not quick enough to pick up the multitude of images, so they were clogging up the memory.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload