Databases
StackOverflow2013
Postsdbo
POReducing numpy memory footprint in long-running application
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POReducing numpy memory footprint in long-running application
    primarykey
    data
    text
    <p>In my application one hundred numpy arrays (1000 complex elements each) are generated and filled with data. Then over many iterations, the array elements are modified over and over again. After the initial generation, the system monitor reports around 50 Mb of RAM usage. Although I am not generating any new arrays, the footprint keeps growing by around 40 Mb per iteration.</p> <p>I learned <a href="https://stackoverflow.com/questions/6018986/memory-profiler-for-numpy">here</a>, that the garbage collector does not handle numpy arrays. So I assume that some temporary arrays I am generating to manipulate data are not collected correctly.</p> <p><a href="https://stackoverflow.com/questions/3372444/profile-memory-allocation-in-python-with-support-for-numpy-arrays?lq=1">Here</a> it says that guppy.hpy().heap() does not help with profiling numpy, unfortunately.</p> <p>How can I identify the source of the problem and ideally keep consumption constant over any number of iterations?</p> <p>I suspect that I may be generating copies when assigning array elements as described <a href="http://docs.scipy.org/doc/numpy/reference/arrays.nditer.html#modifying-array-values" rel="nofollow noreferrer">here</a>, which are then not garbage collected.</p> <p>Can I manually dispose of temporary numpy arrays to assist garbage collection?</p> <p>[Update 1]: Sample code</p> <p>This bit of code is called thousands of times. Each time, the footprint increases. I cannot see why, because to my understanding it is only reading existing arrays and manipulating other existing arrays. Are any of these slicing operations doing something unintended? (Sorry for the line length. I can simplify it, but then I might be hiding my errors, too.)</p> <pre><code>for ts in np.arange(numTimeslots): for fc in np.arange(numFreqChunks): interfencep = np.sum( np.dot(np.dot(self.baseStations[bs].cells[cell].CSI_OFDMA[:,:,fc,ts] ,np.diag(cell.OFDMA_power[:,fc,ts])),self.baseStations[bs].cells[cell].CSI_OFDMA[:,:,fc,ts].conj().T) for bs in self.baseStations for cell in bs.cells if cell != self._cell) noisep = np.eye(self.antennas) * (self.noisePower / numFreqChunks) self.OFDMA_interferenceCovar[:,:,fc,ts] = noisep + interfencep self.OFDMA_EC[:,:,fc,ts] = (np.dot(np.dot(self.OFDMA_CSI[:,:,fc,ts],linalg.inv(noisep+interfencep)),self.OFDMA_CSI[:,:,fc,ts].conj().T)) eigs = linalg.eig(self.OFDMA_EC[:,:,fc,ts])[0] self.OFDMA_SINR[:,fc,ts] = np.real(eigs) </code></pre> <p>[Update 2]: For those curious, this is part of a mobile network simulator. Running on virtualenv, Python 2.7.3, Numpy 1.6.2, SciPy 0.11.0b1</p> <p>[Update 3]: Via commenting it and checking the system monitor, I can identify the 'interferencep = ...'-line as a culprit. It allocated significant memory which is not freed. But why?</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USHauke
    plurals
    1. PL
      singulars
      1. LTLinked
    2. PL
      singulars
      1. LTLinked
    3. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POReducing numpy memory footprint in long-running application
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POReducing numpy memory footprint in long-running application
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POReducing numpy memory footprint in long-running application
      1. This table or related slice is empty.
      1. VTModeratorReview
    1. COCan you reproduce the issue with a simple code example?
      singulars
      1. POReducing numpy memory footprint in long-running application
      1. USJanne Karila
    2. CONo, unfortunately I haven't been able to narrow it down within the program.
      singulars
      1. POReducing numpy memory footprint in long-running application
      1. USHauke
    3. COWhat data type are the elements?
      singulars
      1. POReducing numpy memory footprint in long-running application
      1. USBitwise
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload