Databases
StackOverflow2013
Postsdbo
POMemory leak in tornado generator engine with try/finally block when connections are closed
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POMemory leak in tornado generator engine with try/finally block when connections are closed
    primarykey
    data
    text
    <p>This awesome code, shows memory leak in tornado's <code>gen</code> module, when connections are closed without reading the response:</p> <pre><code>import gc from tornado import web, ioloop, gen class MainHandler(web.RequestHandler): @web.asynchronous @gen.engine def get(self): gc.collect() print len(gc.garbage) # print zombie objects count self.a = '*' * 500000000 # ~500MB data CHUNK_COUNT = 100 try: for i in xrange(CHUNK_COUNT): self.write('*' * 10000) # write ~10KB of data yield gen.Task(self.flush) # wait for reciever to recieve print 'finished' finally: print 'finally' application = web.Application([ (r"/", MainHandler), ]) application.listen(8888) ioloop.IOLoop.instance().start()</code></pre> <p>and now, run a simple test client, multiple times</p> <pre><code>#!/usr/bin/python import urllib urlopen('http://127.0.0.1:8888/') # exit without reading response </code></pre> <p>Now, server output shows, incremental memory usage:</p> <pre><code>0 WARNING:root:Write error on 8: [Errno 104] Connection reset by peer 1 WARNING:root:Read error on 8: [Errno 104] Connection reset by peer WARNING:root:error on read Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/iostream.py", line 361, in _handle_read if self._read_to_buffer() == 0: File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/iostream.py", line 428, in _read_to_buffer chunk = self._read_from_socket() File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/iostream.py", line 409, in _read_from_socket chunk = self.socket.recv(self.read_chunk_size) error: [Errno 104] Connection reset by peer 2 ERROR:root:Uncaught exception GET / (127.0.0.1) HTTPRequest(protocol='http', host='127.0.0.1:8888', method='GET', uri='/', version='HTTP/1.0', remote_ip='127.0.0.1', body='', headers={'Host': '127.0.0.1:8888', 'User-Agent': 'Python-urllib/1.17'}) Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/web.py", line 1021, in _stack_context_handle_exception raise_exc_info((type, value, traceback)) File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/web.py", line 1139, in wrapper return method(self, *args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/gen.py", line 120, in wrapper runner.run() File "/usr/local/lib/python2.7/dist-packages/tornado-2.4.1-py2.7.egg/tornado/gen.py", line 345, in run yielded = self.gen.send(next) File "test.py", line 10, in get self.a = '*' * 500000000 MemoryError ERROR:root:500 GET / (127.0.0.1) 3.91ms </code></pre> <p>If you set <code>CHUNK_COUNT</code> to 1, the 10KB of data can be written to OS connection buffer, and 'finished' and 'finally' texts will be printed to console, and because generator is completed, no memory leak occurs.</p> <p><strong>But the strange part</strong> is that if your remove the try/finally block, the problem disappears!! (even with <code>CHUNK_COUNT</code> set to 100)</p> <p>Is this a bug on CPython or tornado or ...?!</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USTaha Jahangir
    1. USTaha Jahangir
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POMemory leak in tornado generator engine with try/finally block when connections are closed
      1. USsberry
      1. VTFavorite
    2. VO
      singulars
      1. POMemory leak in tornado generator engine with try/finally block when connections are closed
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POMemory leak in tornado generator engine with try/finally block when connections are closed
      1. This table or related slice is empty.
      1. VTUpMod
    1. COwhat version of tornado are you using?
      singulars
      1. POMemory leak in tornado generator engine with try/finally block when connections are closed
      1. USCrackerJack9
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload