Databases
StackOverflow2013
Postsdbo
POTornado "error: [Errno 24] Too many open files" error
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POTornado "error: [Errno 24] Too many open files" error
    primarykey
    data
    text
    <p>I've worked with Tornado quite a bit, but this is the first time I've run into this sort of error. I've been working on a very basic URL shortener. URLs are put into the database by a different application, this one just reads the URLs from a MongoDB store and redirects the clients. After I'd written the basic code I set up a simple 'Siege' test against it, after about 30 seconds of running siege (run with <code>siege -c 64 -t 5m -r 1 http://example.com/MKy</code> against 4 application threads) I started getting 500 responses. Looking in the error log I saw this;</p> <pre><code>ERROR:root:500 GET /MKy (127.0.0.1) 2.05ms ERROR:root:Exception in I/O handler for fd 4 Traceback (most recent call last): File "/opt/python2.7/lib/python2.7/site-packages/tornado-2.1-py2.7.egg/tornado/ioloop.py", line 309, in start File "/opt/python2.7/lib/python2.7/site-packages/tornado-2.1-py2.7.egg/tornado/netutil.py", line 314, in accept_handler File "/opt/python2.7/lib/python2.7/socket.py", line 200, in accept error: [Errno 24] Too many open files ERROR:root:Uncaught exception GET /MKy (127.0.0.1) HTTPRequest(protocol='http', host='shortener', method='GET', uri='/MKy', version='HTTP/1.0', remote_ip='127.0.0.1', body='', headers={'Host': 'shortener', 'Accept-Encoding': 'gzip', 'X-Real-Ip': '94.23.155.32', 'X-Forwarded-For': '94.23.155.32', 'Connection': 'close', 'Accept': '*/*', 'User-Agent': 'JoeDog/1.00 [en] (X11; I; Siege 2.66)'}) Traceback (most recent call last): File "/opt/python2.7/lib/python2.7/site-packages/tornado-2.1-py2.7.egg/tornado/web.py", line 1040, in wrapper File "main.py", line 58, in get File "main.py", line 21, in dbmongo File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 349, in __init__ File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 510, in __find_master File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 516, in __try_node File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/database.py", line 301, in command File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/collection.py", line 441, in find_one File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/cursor.py", line 539, in loop File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/cursor.py", line 560, in _refresh File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/cursor.py", line 620, in __send_message File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 735, in _send_message_with_response File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 591, in __stream File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 200, in get_stream File "/opt/python2.7/lib/python2.7/site-packages/apymongo-0.0.1-py2.7-linux-x86_64.egg/apymongo/connection.py", line 559, in __connect AutoReconnect: could not connect to [('127.0.0.1', 27017)] </code></pre> <p>Important (I guess);</p> <blockquote> <p>error: [Errno 24] Too many open files</p> </blockquote> <p>The code; (It's very simple)</p> <pre><code>import tornado.ioloop import tornado.web import tornado.escape import apymongo import time import sys #Useful stuff (Connect to Mongo) class setup(tornado.web.RequestHandler): def dbmongo(self): if not hasattr(self, '_dbmongo'): self._dbmongo = apymongo.Connection("127.0.0.1", 27017) return self._dbmongo #Basic method to lookup URLs from Mongo and redirect accordingly class expand(setup): @tornado.web.asynchronous def get(self, url): self.mongo = self.dbmongo() #Lookup the URL cursor = self.mongo.rmgshortlinks.links.find_one({'short':url}, self.direct) def direct(self, response): if response == None: self.send_error(404) self.finish() return link = tornado.escape.url_unescape(response['long']) #Bounce the client self.write("&lt;!DOCTYPE html&gt;&lt;html&gt;&lt;head&gt;&lt;meta charset=\"UTF-8\" /&gt;&lt;meta http-equiv=\"refresh\" content=\"0;URL="+link+"\"&lt;/head&gt;&lt;body&gt;&lt;a href=\""+link+"\"&gt;Click Here&lt;/a&gt;&lt;/body&gt;&lt;/html&gt;") self.finish(); #Define the URL routes application = tornado.web.Application([ (r"/([a-zA-Z0-9]+)", expand) ]) #Start the server if __name__ == "__main__": listening_port = int(sys.argv[1]) if listening_port &gt; 0: application.listen(listening_port) tornado.ioloop.IOLoop.instance().start() else: sys.stderr.write("No port specified!") </code></pre> <p>The dev server I'm using has 8 cores and 64GB memory, running RedHat Enterprise Linux 5 and Python 2.6. I've never had these sorts of issues with Tornado/Async Mongo applications before.</p> <p>Probably useful information;</p> <pre><code>[root@puma ~]# ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 31374 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 1024 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 31374 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited </code></pre> <p>(open files is only set to 1024 but I would have thought that's more than enough)</p> <p>Is Tornado/Apymongo not closing the connections properly? The applications sit behind NGINX but connect using HTTP, Apymongo should be connecting via TCP but might be using sockets. Even so it should be sharing/pooling connections shouldn't it?</p> <h1>Edit</h1> <p>As suggested, moved the app onto one of our testing servers with a max open files limit of 61440, same error after about 30 seconds of running in siege.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USSmudge
    1. USSmudge
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POTornado "error: [Errno 24] Too many open files" error
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POTornado "error: [Errno 24] Too many open files" error
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POTornado "error: [Errno 24] Too many open files" error
      1. USMK Yung
      1. VTFavorite
    1. COAs an aside: if possible, don't send HTML with a meta refresh tag. The right way to redirect a request is with an HTTP 301 or 302 response.
      singulars
      1. POTornado "error: [Errno 24] Too many open files" error
      1. USThomas K
    2. CO@ThomasK There are many reasons we're doing it that way, the main one being it preserves/overwrites the refer header. Twitter's t.co does a similar thing depending on the client (Try CURL-ing a t.co URL with a real browser header)
      singulars
      1. POTornado "error: [Errno 24] Too many open files" error
      1. USSmudge
    3. CODid you try the solution found here: http://stackoverflow.com/questions/2569620/socket-accept-error-24-to-many-open-files
      singulars
      1. POTornado "error: [Errno 24] Too many open files" error
      1. USAlvin K.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload