Databases
StackOverflow2013
Postsdbo
POProfiling Netty Performance
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POProfiling Netty Performance
    primarykey
    data
    text
    <p>I'm writing a Netty application. The application is running on a 64 bit eight core linux box</p> <p>The Netty application is a simple router that accepts requests (incoming pipeline) reads some metadata from the request and forwards the data to a remote service (outgoing pipeline).</p> <p>This remote service will return one or more responses to the outgoing pipeline. The Netty application will route the responses back to the originating client (the incoming pipeline)</p> <p>There will be thousands of clients. There will be thousands of remote services.</p> <p>I'm doing some small scale testing (ten clients, ten remotes services) and I don't see the sub 10 millisecond performance I'm expecting at a 99.9 percentile. I'm measuring latency from both client side and server side.</p> <p>I'm using a fully async protocol that is similar to SPDY. I capture the time (I just use System.nanoTime()) when we process the first byte in the FrameDecoder. I stop the timer just before we call channel.write(). I am measuring sub-millisecond time (99.9 percentile) from the incoming pipeline to the outgoing pipeline and vice versa.</p> <p>I also measured the time from the first byte in the FrameDecoder to when a ChannelFutureListener callback was invoked on the (above) message.write(). The time was a high tens of milliseconds (99.9 percentile) but I had trouble convincing myself that this was useful data.</p> <p>My initial thought was that we had some slow clients. I watched channel.isWritable() and logged when this returned false. This method did not return false under normal conditions</p> <p>Some facts:</p> <ul> <li>We are using the NIO factories. We have not customized the worker size</li> <li>We have disabled Nagel (tcpNoDelay=true)</li> <li>We have enabled keep alive (keepAlive=true)</li> <li>CPU is idle 90+% of the time</li> <li>Network is idle</li> <li>The GC (CMS) is being invoked every 100 seconds or so for a very short amount of time</li> </ul> <p>Is there a debugging technique that I could follow to determine why my Netty application is not running as fast as I believe it should?</p> <p>It feels like channel.write() adds the message to a queue and we (application developers using Netty) don't have transparency into this queue. I don't know if the queue is a Netty queue, an OS queue, a network card queue or what. Anyway I'm reviewing examples of existing applications and I don't see any anti-patterns I'm following</p> <p>Thanks for any help/insight</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USJake Carr
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POProfiling Netty Performance
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POProfiling Netty Performance
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POProfiling Netty Performance
      1. This table or related slice is empty.
      1. VTUpMod
    1. COYour benchmarking approach might not be as accurate as you think. Have a look at http://code.google.com/p/caliper/wiki/JavaMicrobenchmarks for some good insight into why.
      singulars
      1. POProfiling Netty Performance
      1. USsworisbreathing
    2. COmaybe I am making a microbenchmark. But I am load testing the application while I am measuring latencies. Also, I'm making lots of measurements (1000 requests per second, 99.9th percentile over ten seconds). Clearly the client experience when the service is under load is unacceptable. How do I dig down in netty and figure out if the additional latency is my code, netty's code, the operating system, the network card, the network, etc? My code is apparently using less than a microsecond of wallclock time (as far as I can tell)
      singulars
      1. POProfiling Netty Performance
      1. USJake Carr
    3. COIs it an open-sourced project? Did you find the issue?
      singulars
      1. POProfiling Netty Performance
      1. USmatanster
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload