Databases
StackOverflow2013
Postsdbo
POJetty hangs with 100% CPU on sun.nio.ch.FileDispatcherImpl.read0(Native Method)
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POJetty hangs with 100% CPU on sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    primarykey
    data
    text
    <p>(update: it has nothing to do with file upload, more details at the bottom)</p> <p>I'm running <strong>jetty-distribution-9.0.6.v20130930</strong> server over <strong>jre-1.7.0-openjdk.x86_64</strong>, running on <strong>EC2 linux</strong> server. It's a backend to a mobile app that receives about 50k requests/day.</p> <p>About once a day, a thread hangs with 100% CPU, until I kill it (after running several hours on 100% CPU). </p> <p>JavaMelody shows the following call stack for the hung thread:</p> <p><img src="https://i.stack.imgur.com/2bE7k.jpg" alt="enter image description here"></p> <p>Jetty HTTP connector definition, from etc/jetty-https.xml:</p> <pre><code>&lt;Call id="httpsConnector" name="addConnector"&gt; &lt;Arg&gt; &lt;New class="org.eclipse.jetty.server.ServerConnector"&gt; &lt;Arg name="server"&gt;&lt;Ref refid="Server" /&gt;&lt;/Arg&gt; &lt;Arg name="factories"&gt; &lt;Array type="org.eclipse.jetty.server.ConnectionFactory"&gt; &lt;Item&gt; &lt;New class="org.eclipse.jetty.server.SslConnectionFactory"&gt; &lt;Arg name="next"&gt;http/1.1&lt;/Arg&gt; &lt;Arg name="sslContextFactory"&gt;&lt;Ref refid="sslContextFactory"/&gt;&lt;/Arg&gt; &lt;/New&gt; &lt;/Item&gt; &lt;Item&gt; &lt;New class="org.eclipse.jetty.server.HttpConnectionFactory"&gt; &lt;Arg name="config"&gt;&lt;Ref refid="sslHttpConfig"/&gt;&lt;/Arg&gt; &lt;/New&gt; &lt;/Item&gt; &lt;/Array&gt; &lt;/Arg&gt; &lt;Set name="host"&gt;&lt;Property name="jetty.host" /&gt;&lt;/Set&gt; &lt;Set name="port"&gt;&lt;Property name="jetty.https.port" default="8443" /&gt;&lt;/Set&gt; &lt;Set name="idleTimeout"&gt;30000&lt;/Set&gt; &lt;/New&gt; &lt;/Arg&gt; &lt;/Call&gt; </code></pre> <p>Jetty HTTPS configuration, from start.ini:</p> <pre><code>jetty.https.port=8443 etc/jetty-https.xml </code></pre> <p>JVM version -</p> <pre><code>java version "1.7.0_45" OpenJDK Runtime Environment (amzn-2.4.3.2.32.amzn1-x86_64 u45-b15) OpenJDK 64-Bit Server VM (build 24.45-b08, mixed mode) </code></pre> <p>a</p> <p><strong>Questions</strong></p> <ol> <li><p>What causes this?</p></li> <li><p>Why timeouts are not triggered to kill these hanging threads?</p></li> <li><p>How do I enable timeout of such scenarios?</p></li> </ol> <p><strong>Update</strong></p> <p>Turns out it has nothing to do with file upload, it sometimes happens also in a different EC2 server (same configuration), and sometimes the two instances hang at <strong>exactly the same time</strong>, with the same call stack. </p> <p>The other server doesn't implement file upload, so this option is ruled out.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USKof
    1. USKof
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. COThe `ChannelEndpoint.fill()` means the server is attempting to read from the connection. (the next line shows that you are also using SSL). So that means your stuck thread is the one for `etc/jetty-https.xml` not `etc/jetty-http.xml`. You didn't paste that configuration.
      singulars
      1. POJetty hangs with 100% CPU on sun.nio.ch.FileDispatcherImpl.read0(Native Method)
      1. USJoakim Erdfelt
    2. COright, replaced.
      singulars
      1. POJetty hangs with 100% CPU on sun.nio.ch.FileDispatcherImpl.read0(Native Method)
      1. USKof
    3. COPlease include the specific versions of Jetty and the JVM
      singulars
      1. POJetty hangs with 100% CPU on sun.nio.ch.FileDispatcherImpl.read0(Native Method)
      1. USJoakim Erdfelt
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload