Databases
StackOverflow2013
Postsdbo
POHow thread safe is thrift? re: I seem to have requests disrupting one another
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow thread safe is thrift? re: I seem to have requests disrupting one another
    text
    copied!<h2>edit</h2> <p>Apparently what I was hoping to do is outside of the scope of thrift... If I make sure there is never more than one client on the port, everything is a-ok. Of course this kind of defeats the purpose as I'd like to have several reusable connections open to the server to improve response times and lower overhead.</p> <p>If anyone has a suggestion of an alternate way to achieve this, it would be appreciated(or if my conclusion is wrong)</p> <h2>Background</h2> <p>I have a multi-component application that is mostly connected up by thrift(mostly java->php connections).</p> <p>So far it all has appeared to be fine, but having introduced a Java->Java connection where the client end is a servlet that can launch hundreds of requests a second.</p> <p>The method being accessed has the following interface:</p> <pre><code>bool pvCheck(1:i32 toolId) throws(1:DPNoToolException nte), </code></pre> <p>To make sure it wasn't something weird on the service end I replaced the implementation with a trivial one:</p> <pre><code> @Override public boolean pvCheck(int toolId) throws TException { //boolean ret = api.getViewsAndDec(toolId); return true; } </code></pre> <h2>The errors/possible causes?</h2> <p>So long as there's not many connections it goes by fine but as soon as connections come close together, connections start getting stuck in the reader.</p> <p>If I pull one of them up in the debugger the stack looks like this: </p> <pre><code>Daemon Thread [http-8080-197] (Suspended) BufferedInputStream.read(byte[], int, int) line: 308 TSocket(TIOStreamTransport).read(byte[], int, int) line: 126 TSocket(TTransport).readAll(byte[], int, int) line: 84 TBinaryProtocol.readAll(byte[], int, int) line: 314 TBinaryProtocol.readI32() line: 262 TBinaryProtocol.readMessageBegin() line: 192 DumboPayment$Client.recv_pvCheck() line: 120 DumboPayment$Client.pvCheck(int) line: 105 Receiver.performTask(HttpServletRequest, HttpServletResponse) line: 157 Receiver.doGet(HttpServletRequest, HttpServletResponse) line: 109 Receiver(HttpServlet).service(HttpServletRequest, HttpServletResponse) line: 617 Receiver(HttpServlet).service(ServletRequest, ServletResponse) line: 717 ApplicationFilterChain.internalDoFilter(ServletRequest, ServletResponse) line: 290 ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 206 StandardWrapperValve.invoke(Request, Response) line: 233 StandardContextValve.invoke(Request, Response) line: 191 StandardHostValve.invoke(Request, Response) line: 127 ErrorReportValve.invoke(Request, Response) line: 102 StandardEngineValve.invoke(Request, Response) line: 109 CoyoteAdapter.service(Request, Response) line: 298 Http11AprProcessor.process(long) line: 859 Http11AprProtocol$Http11ConnectionHandler.process(long) line: 579 AprEndpoint$Worker.run() line: 1555 Thread.run() line: 619 </code></pre> <p>This seems to be triggered by data getting corrupted as I get the following exceptions:</p> <pre><code>10/11/22 18:38:55 WARN logger.Receiver: pvCheck had an exception org.apache.thrift.TApplicationException: pvCheck failed: unknown result at *.thrift.generated.DumboPayment$Client.recv_pvCheck(DumboPayment.java:135) at *.thrift.generated.DumboPayment$Client.pvCheck(DumboPayment.java:105) at *.Receiver.performTask(Receiver.java:157) at *.Receiver.doGet(Receiver.java:109) at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11AprProcessor.process(Http11AprProcessor.java:859) at org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(Http11AprProtocol.java:579) at org.apache.tomcat.util.net.AprEndpoint$Worker.run(AprEndpoint.java:1555) at java.lang.Thread.run(Thread.java:619) </code></pre> <p>and</p> <pre><code>10/11/22 17:59:46 ERROR [/ninja_ar].[Receiver]: サーブレット Receiver のServlet.service()が例外を投げました java.lang.OutOfMemoryError: Java heap space at org.apache.thrift.protocol.TBinaryProtocol.readStringBody(TBinaryProtocol.java:296) at org.apache.thrift.protocol.TBinaryProtocol.readString(TBinaryProtocol.java:290) at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:198) at *.thrift.generated.DumboPayment$Client.recv_pvCheck(DumboPayment.java:120) at *.thrift.generated.DumboPayment$Client.pvCheck(DumboPayment.java:105) at *.Receiver.performTask(Receiver.java:157) at *.Receiver.doGet(Receiver.java:109) at javax.servlet.http.HttpServlet.service(HttpServlet.java:690) at javax.servlet.http.HttpServlet.service(HttpServlet.java:803) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:172) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870) at org.apache.coyote.http11.Http11BaseProtocol$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665) at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528) at org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81) at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:685) at java.lang.Thread.run(Thread.java:636) </code></pre> <p>Perhaps I'm way off the mark but I'm pretty sure these are related to the client continuing to attempt to read when there's nothing being sent.</p> <h2>Some implementation details</h2> <p>Both the server and client are using the java binary protocol.</p> <p>I wrote a simple client pool class which lets me reuse clients, these are the main functions:</p> <pre><code>public synchronized Client getClient() { if(clientQueue.isEmpty()) { return newClient(); } else { return clientQueue.getLast(); } } private synchronized Client newClient() { int leftToTry = serverArr.length; Client cli = null; while(leftToTry &gt; 0 &amp;&amp; cli == null) { log.info("Creating new connection to " + serverArr[roundRobinPos] + port); TTransport transport = new TSocket(serverArr[roundRobinPos], port); TProtocol protocol = new TBinaryProtocol(transport); cli = new Client(protocol); try { transport.open(); } catch (TTransportException e) { cli = null; log.warn("Failed connection to " + serverArr[roundRobinPos] + port); } roundRobinPos++; if(roundRobinPos &gt;= serverArr.length) { roundRobinPos = 0; } leftToTry--; } return cli; } public void returnClient(Client cli) { clientQueue.addFirst(cli); } </code></pre> <p>The client applications(namely tomcat servlets) access it in the following way:</p> <pre><code> Client dpayClient = null; if(dpay != null &amp;&amp; (dpayClient = dpay.getClient()) != null) { try { dpayClient.pvCheck(requestParameters.getId()); } catch (DPNoToolException e) { return; } catch (TException e) { log.warn("pvCheck had an exception", e); } finally { if(dpayClient != null) { dpay.returnClient(dpayClient); } } } </code></pre> <p>The actual thrift connection is upped in the following manner</p> <pre><code>private boolean initThrift(int port, Configuration conf) { TProtocolFactory protocolFactory = new TBinaryProtocol.Factory(); DPaymentHandler handler = new DPaymentHandler(conf); DumboPayment.Processor processor = new DumboPayment.Processor(handler); InetAddress listenAddress; try { listenAddress = InetAddress.getLocalHost(); } catch (UnknownHostException e) { LOG.error("Failed in thrift init", e); return false; } TServerTransport serverTransport; try { serverTransport = new TServerSocket( new InetSocketAddress(listenAddress, port)); } catch (TTransportException e) { LOG.error("Failed in thrift init", e); return false; } TTransportFactory transportFactory = new TTransportFactory(); TServer server = new TThreadPoolServer(processor, serverTransport, transportFactory, protocolFactory); LOG.info("Starting Dumbo Payment thrift server on " + listenAddress + ":" + Integer.toString(port)); server.serve(); return true; } </code></pre> <h2>Finally</h2> <p>Been stuck on this for a while now... It may well be I'm missing something obvious. I'd really appreciate any help with this.</p> <p>If any additional information is needed, please let me know. There's a whole mouthful there, so I wanted to try to keep stuff to the most (hopefully) relevant.</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload