Databases
StackOverflow2013
Postsdbo
PONode.js struggling with lots of concurrent connections
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PONode.js struggling with lots of concurrent connections
    primarykey
    data
    text
    <p>I'm working on a somewhat unusual application where 10k clients are precisely timed to all try to submit data at once, every 3 mins or so. This 'ab' command fairly accurately simulates one barrage in the real world:</p> <pre><code>ab -c 10000 -n 10000 -r "http://example.com/submit?data=foo" </code></pre> <p>I'm using Node.js on Ubuntu 12.4 on a rackspacecloud VPS instance to collect these submissions, however, I'm seeing some very odd behavior from Node, even when I remove all my business logic and turn the http request into a no-op.</p> <p>When the test gets about 90% done, it hangs for a long period of time. Strangely, this happens consistently at 90% - for c=n=10k, at 9000; for c=n=5k, at 4500; for c=n=2k, at 1800. The test actually completes eventually, often with no errors. But both ab and node logs show continuous processing up till around 80-90% of the test run, then a long pause before completing.</p> <p>When node is processing requests normally, CPU usage is typically around 50-70%. During the hang period, CPU goes up to 100%. Sometimes it stays near 0. Between the erratic CPU response and the fact that it seems unrelated to the actual number of connections (only the % complete), I do not suspect the garbage collector.</p> <p>I've tried this running 'ab' on localhost and on a remote server - same effect.</p> <p>I suspect something related to the TCP stack, possibly involving closing connections, but none of my configuration changes have helped. My changes:</p> <ul> <li>ulimit -n 999999</li> <li>When I listen(), I set the backlog to 10000</li> </ul> <p>Sysctl changes are:</p> <pre><code>net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 65536 16777216 net.ipv4.tcp_max_orphans = 20000 net.ipv4.tcp_max_syn_backlog = 10000 net.core.somaxconn = 10000 net.core.netdev_max_backlog = 10000 </code></pre> <p>I have also noticed that I tend to get this msg in the kernel logs:</p> <pre><code>TCP: Possible SYN flooding on port 80. Sending cookies. Check SNMP counters. </code></pre> <p>I'm puzzled by this msg since the TCP backlog queue should be deep enough to never overflow. If I disable syn cookies the "Sending cookies" goes to "Dropping connections".</p> <p>I speculate that this is some sort of linux TCP stack tuning problem and I've read just about everything I could find on the net. Nothing I have tried seems to matter. Any advice?</p> <p><strong>Update:</strong> Tried with tcp_max_syn_backlog, somaxconn, netdev_max_backlog, and the listen() backlog param set to 50k with no change in behavior. Still produces the SYN flood warning, too. </p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USstickfigure
    1. USstickfigure
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. PONode.js struggling with lots of concurrent connections
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PONode.js struggling with lots of concurrent connections
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PONode.js struggling with lots of concurrent connections
      1. USAndrey.Dankevich
      1. VTFavorite
    1. COOn a side note I am very interested in the final solution to this as I need nodejs with high number of active connections also.
      singulars
      1. PONode.js struggling with lots of concurrent connections
      1. USAndrew T Finnell
    2. COFor what it's worth, as efficient as Node can be, 10k connections all doing something at once is more load than I would leave to a single VPS.
      singulars
      1. PONode.js struggling with lots of concurrent connections
      1. USBrad
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload