Databases
StackOverflow2013
Postsdbo
PONode.js+Express Randomly Drops Requests, resulting in a Gateway Timeout
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PONode.js+Express Randomly Drops Requests, resulting in a Gateway Timeout
    primarykey
    data
    text
    <p><strong>EDIT</strong></p> <p>After much putzing around, I finally found something that seems like it might be a solid lead:</p> <p>The express library fails to accept the incoming request when it is currently using the Node+OAuth module to perform several outbound requests (eg, to Facebook, Twitter, etc.). I was able to determine this by putting lots of logs throughout my code, where I discovered that the "begin-request" log (described below) was not triggered when in the middle of an outbound request.</p> <p>I have been able to demonstrably show that when the Node+OAuth module is making a few outbound requests, the inbound requests to my API (via a browser window) will hang and not be received until one of those outbound OAuth requests has finished.</p> <p>Of course, I've already done:</p> <pre><code>require('http').globalAgent.maxSockets = 999; </code></pre> <p>Per a suggestion in IRC, I've added</p> <pre><code>console.log(require('http').globalAgent.requests); </code></pre> <p>But this always seems to be === {}, implying that there are no pending inbound requests AFAIK.</p> <p>Thus I am left to conclude that either node.js or express is choosing to, for some reason, block incoming requests due to the outbound requests, even though there should be plenty of sockets available...</p> <p>Anybody have any hints on how to solve this?</p> <hr> <p>I have an API created in node.js using Express, Mongoose, etc. deployed on the Amazon Cloud which works wonderfully and fast 99% of the time.</p> <p>Except, once in a while, a request seems to be somehow dropped or otherwise ignored. I'm talking about requests that usually complete in milliseconds randomly being unresponsive with no clear picture <strong>why</strong>.</p> <p>The symptom is a simple "Gateway Timeout" when connecting to the API endpoint. An identical request, made from the same client with the same params, just moments before or moments after, will work just fine.</p> <p>Of course, my first thought was "duh, server overload!" So I spent a lot of time optimizing my requests, monogoDB, etc. Finally I got to the point that the CPU/disk/RAM usage across the board (both in Node.js servers and in Mongo servers) was <strong>very</strong> low. I use Scout and RightScale to track my servers in real-time, and I log any request or query that takes more than 100ms. My node servers currently have 5GB of FREE RAM, 70% Free CPU (on the 1st core), etc. So I'm 99.99% certain it is not a performance issue.</p> <p>Finally, I fell back on a desperate attempt: I attached a random number to all requests being made by my client(s). Then, in the node.js app, I do a console.log() when the request is first received and when it completes. Eg, here's the middleware I use in express:</p> <pre><code>var configureAPI = function() { return function(req, res, next) { if(req.body.ruid) console.log(req.body.ruid); // more middleware stuff... }; } server.configure(function(){ server.use(express.bodyParser()); server.use(configureAPI()); server.use(onError); // ... more config stuff } </code></pre> <p>What I found shocked me: apparently, <strong>the node.js app is not even receiving the requests in question</strong>. I have a Javascript webapp, and I print the "ruid" sent with the request to the console. Whenever the request succeeds, there is a corresponding "ruid" printed in the node.js console. Whenever it times out, there is not.</p> <hr> <p>Edit: more debugging &amp; info.</p> <p>My app servers actually started (and continue) to also serve PHP (thus, they have Apache, etc. installed). I needed <a href="http://streamified.me" rel="noreferrer">http://streamified.me</a> to serve my website (PHP) and <a href="http://api.streamified.me" rel="noreferrer">http://api.streamified.me</a> to serve my API (node.js)... so I have a line in my httpd.conf file to cause requests to api.streamified.me (instead of streamified.me) to go to node.js via port 8888:</p> <pre><code>RewriteCond %{HTTP_HOST} ^api.streamified.me RewriteRule ^(.*) http://localhost:8888$1 [P] </code></pre> <p>So, in the same httpd.conf file, I turned on RewriteLogLevel 5 and then created a simple PHP+CURL script on my localhost to hit my api.streamified.me with a random URL (which should cause node.js to trigger a simple "not found" response) until it resulted in a Gateway timeout. Here, you can see that it has happened -- and the rewrite log shows that the request was definitely received by the app server and forwarded to port 8888... but it was never received by node.js (or, at least, the first line of code in the first line of middleware never gets it...)</p> <p><img src="https://i.stack.imgur.com/3OQxS.png" alt="enter image description here"></p> <hr> <p>I've been over and over my node.js code and am quite certain I do not have blocking code, and even if I did, I cannot imagine it blocking the thread long enough to miss a request without it raising a red flag somewhere.</p> <p>What am I missing? Is there some reason the incoming socket would be blocked? I do make a fair number of HTTP requests to external APIs via my node.js app, but AFAIK that should not be blocking incoming sockets.</p> <hr> <p>Of course, I have error logging in place. I have enabled it on the process level...</p> <pre><code>process.addListener("uncaughtException", function (err) { // some logging code } </code></pre> <p>and on the Express level (the onError handler above). I know that my error logging functions work, because I've seen them both fire before. But neither of them report anything around the time of the dropped requests, nor do I see anything in the console...</p> <hr> <ul> <li>Express Version: 3.0.0rc5</li> <li>Node.js version: 0.8.12</li> <li>2 instances of the node.js app running on a standard Amazon Cloud setup (m1.large instances), behind 2 Load Balancers, connecting to a 3x replica set of MongoDBs (also m1.large)</li> </ul>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USZane Claes
    1. USZane Claes
    plurals
    1. This table or related slice is empty.
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. PONode.js+Express Randomly Drops Requests, resulting in a Gateway Timeout
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PONode.js+Express Randomly Drops Requests, resulting in a Gateway Timeout
      1. USFutur
      1. VTFavorite
    3. VO
      singulars
      1. PONode.js+Express Randomly Drops Requests, resulting in a Gateway Timeout
      1. This table or related slice is empty.
      1. VTUpMod
    1. COYou've confirmed that the load balancers are receiving the request and sending it to your node servers successfully? How frequently are you making requests when one fails?
      singulars
      1. PONode.js+Express Randomly Drops Requests, resulting in a Gateway Timeout
      1. USBill
    2. COThe same LBs / app servers also serve PHP files, which never result in a timeout. I'm not quite sure how to confirm that the LBs are forwarding to the Node servers correctly, though, aside from that. I'm not showing any spikes in traffic; apache logs on Rightscale report a consistent ~10 req/sec.
      singulars
      1. PONode.js+Express Randomly Drops Requests, resulting in a Gateway Timeout
      1. USZane Claes
    3. COI found a couple of bugs listed that described similar issues but they were all fixed by 0.6.6. You might try upgrading to the latest version since there have been a ton of fixes/improvements since 0.6. I'd also suggest you setup a network sniffer on your app servers to make sure the servers are actually receiving the packets.
      singulars
      1. PONode.js+Express Randomly Drops Requests, resulting in a Gateway Timeout
      1. USBill
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload