Databases
StackOverflow2013
Postsdbo
POSocket.read() thread hanging between JBoss and ActiveMQ
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POSocket.read() thread hanging between JBoss and ActiveMQ
    primarykey
    data
    text
    Given <ul> <li>My Java app is a WAR deployed to JBoss (4.0.4GA)</li> <li>Publishes and subscribes to an ActiveMQ (5.6.0) instance</li> <li>Java app uses Apache Camel (2.10.3) for all integration (producing &amp; consuming) with ActiveMQ</li> <li>JBoss and ActiveMQ on their own (CentOS 5.6 Final) quad-core virtual servers, each virtual is on a different physical</li> </ul> <p>I have a thread-hanging issue and am seeing the following in my thread dump:</p> <pre><code>java.net.SocketInputStream.socketRead0(Native Method) java.net.SocketInputStream.read(SocketInputStream.java:129) java.io.BufferedInputStream.fill(BufferedInputStream.java:218) java.io.BufferedInputStream.read1(BufferedInputStream.java:258) java.io.BufferedInputStream.read(BufferedInputStream.java:317) sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:687) sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:632) sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1195) java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:379) org.springframework.remoting.httpinvoker.SimpleHttpInvokerRequestExecutor.validateResponse(SimpleHttpInvokerRequestExecutor.java:146) org.springframework.remoting.httpinvoker.SimpleHttpInvokerRequestExecutor.doExecuteRequest(SimpleHttpInvokerRequestExecutor.java:66) org.springframework.remoting.httpinvoker.AbstractHttpInvokerRequestExecutor.executeRequest(AbstractHttpInvokerRequestExecutor.java:136) org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:192) org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.executeRequest(HttpInvokerClientInterceptor.java:174) org.springframework.remoting.httpinvoker.HttpInvokerClientInterceptor.invoke(HttpInvokerClientInterceptor.java:142) org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) $Proxy117.SigmaCruxer(Unknown Source) com.tms.SigmaClient.SigmaClient.processMessage(SigmaClient.java:46) com.tms.SigmaClient.SigmaServiceEndpoint.doSigma(SigmaServiceEndpoint.java:29) com.tms.SigmaClient.SigmaServiceEndpoint.mark(SigmaServiceEndpoint.java:43) sun.reflect.GeneratedMethodAccessor193.invoke(Unknown Source) sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) java.lang.reflect.Method.invoke(Method.java:597) org.apache.camel.component.bean.MethodInfo.invoke(MethodInfo.java:329) org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:231) org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:169) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:104) org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:74) org.apache.camel.impl.ProcessorEndpoint.onExchange(ProcessorEndpoint.java:102) org.apache.camel.impl.ProcessorEndpoint$1.process(ProcessorEndpoint.java:72) org.apache.camel.impl.converter.AsyncProcessorTypeConverter$ProcessorToAsyncProcessorBridge.process(AsyncProcessorTypeConverter.java:50) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:78) org.apache.camel.processor.SendProcessor$2.doInAsyncProducer(SendProcessor.java:114) org.apache.camel.impl.ProducerCache.doInAsyncProducer(ProducerCache.java:284) org.apache.camel.processor.SendProcessor.process(SendProcessor.java:109) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:78) org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98) org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89) org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:69) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:78) org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98) org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89) org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:99) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:78) org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:318) org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:209) org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:305) org.apache.camel.processor.UnitOfWorkProcessor.process(UnitOfWorkProcessor.java:102) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:78) org.apache.camel.processor.DelegateAsyncProcessor.processNext(DelegateAsyncProcessor.java:98) org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:89) org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:69) org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:104) org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:85) org.apache.camel.component.jms.EndpointMessageListener.onMessage(EndpointMessageListener.java:91) org.springframework.jms.listener.AbstractMessageListenerContainer.doInvokeListener(AbstractMessageListenerContainer.java:560) org.springframework.jms.listener.AbstractMessageListenerContainer.invokeListener(AbstractMessageListenerContainer.java:498) org.springframework.jms.listener.AbstractMessageListenerContainer.doExecuteListener(AbstractMessageListenerContainer.java:467) org.springframework.jms.listener.AbstractPollingMessageListenerContainer.doReceiveAndExecute(AbstractPollingMessageListenerContainer.java:325) org.springframework.jms.listener.AbstractPollingMessageListenerContainer.receiveAndExecute(AbstractPollingMessageListenerContainer.java:263) org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.invokeListener(DefaultMessageListenerContainer.java:1058) org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.executeOngoingLoop(DefaultMessageListenerContainer.java:1050) org.springframework.jms.listener.DefaultMessageListenerContainer$AsyncMessageListenerInvoker.run(DefaultMessageListenerContainer.java:947) java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) java.lang.Thread.run(Thread.java:662) </code></pre> <p>According to these two artices: (<a href="http://javaeesupportpatterns.blogspot.com/2011/08/javanetsocketinputstreamsocketread0.html">here</a> and <a href="http://javaeesupportpatterns.blogspot.com/2011/04/javanetsocketinputstreamsocketread0.html">here</a>), my JBoss app has a blocking I/O operation on <code>Socket.read()</code> that is waiting for a completed response from a downstream service provider (in my case, ActiveMQ). Again, according to these articles, the culprit is one of the following 3 items:</p> <ul> <li>ActiveMQ is in an unhealthy/unstable state and is responding too slowly, causing my listening/waiting/blocking threads to hang; or</li> <li>The ActiveMQ instance itself is fine, but is processing an operation (writing to KahaDB, etc.) that is taking too long to complete, again causing my threads to hang; or</li> <li>There are networking issues between my JBoss app (WAR) and my ActiveMQ instance.</li> </ul> <p>I'm trying to figure out which of the three is the case. Is there anything in that thread dump to indicate which one it is? My <em>understanding</em> (after reading those articles) is that the <em>real</em> hang is the fact that the client-side (blocking) socket has just not received all the bytes it needs to consider the response complete; meaning it hasn't received <em>any</em> response from ActiveMQ, or it just hasn't received a full response.</p> <p>So I ask:</p> <ol> <li>Is there a clear indication of which of the 3 scenarios is the case? If so, what/why? If not, what <em>should</em> my next step be (I am also the "admin" who set up ActiveMQ so I have full access to it as well as JBoss and the WAR deployed to it).</li> <li>Would upgrading to a newer of JBoss fix this? Perhaps 4.0.4GA is using the "old" (blocking) Java I/O, whereas newer versions might use NIO? Probably a long-shot but can't discredit it just yet.</li> <li>Both articles stress that proper socket-timeout configuration should be implemented which may very well mitigate all of this (although it doesn't address the underlying ActiveMQ unresponsiveness and/or networking issues): <ol> <li>Is this a timeout I would write in my Java code? If so how and with what API? JMS? Some ActiveMQ client-side jar?</li> <li>Is this a timeout I implement at the OS level? If so I'm not sure how to proceed...</li> <li>Is this a timeout I implement on the server-side (ActiveMQ)? If so, how?</li> </ol></li> </ol> <p>I think I'm closing in on a solution here, but kind of stuck and having a tough time seeing the forest through the trees. Thanks in advance!</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USIAmYourFaja
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POSocket.read() thread hanging between JBoss and ActiveMQ
      1. USIAmYourFaja
      1. VTBountyStart
    2. VO
      singulars
      1. POSocket.read() thread hanging between JBoss and ActiveMQ
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POSocket.read() thread hanging between JBoss and ActiveMQ
      1. This table or related slice is empty.
      1. VTUpMod
    1. COFrom the stack trace whatever is running camel has already received a message from activemq and that message has been dispatched to the listener in camel (the onMessage call). Camel has done some logic and is making a SigmaCruxer call, which looks like a web service call. It is that call that is blocked on the socket read in the stack trace. What is that web service call attempting to contact and is that alive and responsive? That doesn't appear to be an activemq endpoint - are you sure that your socket is blocked at activemq?
      singulars
      1. POSocket.read() thread hanging between JBoss and ActiveMQ
      1. USphilwb
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload