Databases
StackOverflow2013
Postsdbo
POconfused about httpclient performance
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POconfused about httpclient performance
    primarykey
    data
    text
    <p>I'm just getting started writing a simple web crawler to get info on links we have coming in to our system. I'm using httpclient 4.x. I have about 100 threads running fetching links and doing head requests on them, it works great for the first few hours then it slows to a screeching crawl. I'm not sure if I'm setting up the connection manager properly or not.</p> <p>here is the code I have to create an httpclient object. Anyone see anything that would raise an alarm with this code block? When I stop the server and restart it everything is as good as new again. During the phase when it's running slow the memory still looks ok at a steady 500K per process so it doesn't look like I'm leaking memory.</p> <pre><code>HttpParams httpParams = new BasicHttpParams(); HttpConnectionParams.setConnectionTimeout(httpParams, 5000); HttpConnectionParams.setSoTimeout(httpParams, 5000); ConnManagerParams.setMaxTotalConnections(httpParams, 200); HttpProtocolParams.setVersion(httpParams, HttpVersion.HTTP_1_1); // set request params httpParams.setParameter("http.protocol.cookie-policy", CookiePolicy.BROWSER_COMPATIBILITY); httpParams.setParameter("http.useragent", "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)"); SchemeRegistry schemeRegistry = new SchemeRegistry(); schemeRegistry.register(new Scheme("http", PlainSocketFactory.getSocketFactory(), 80)); schemeRegistry.register(new Scheme("https", PlainSocketFactory.getSocketFactory(), 443)); final ClientConnectionManager cm = new ThreadSafeClientConnManager(httpParams,schemeRegistry); HttpClient httpClient = new DefaultHttpClient(cm, httpParams); httpClient.getParams().setParameter("http.conn-manager.timeout", 10000L); httpClient.getParams().setParameter("http.protocol.wait-for-continue", 10000L); </code></pre> <p>I'm also using this code in a thread to clean up expired connections as mentioned in the docs</p> <pre><code>final Runnable cleanUp = new Runnable() { public void run() { cm.closeExpiredConnections(); // Optionally, close connections // that have been idle longer than 30 sec cm.closeIdleConnections(30, TimeUnit.SECONDS); } }; </code></pre> <p>UPDATE: I ran visual VM for an hour or so and here's the memory graph on the remote process, the memory is now used up</p> <p><a href="http://img64.imageshack.us/f/screenshot20100714at204.png/" rel="nofollow noreferrer">http://img64.imageshack.us/f/screenshot20100714at204.png/</a></p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USJames
    1. USJames
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. COI know you said memory looks stable, but did you enable GC logging? Are you spending any significant time in GC?
      singulars
      1. POconfused about httpclient performance
      1. USbwawok
    2. COI did not actually, pretty new to java so not sure where I might go about setting that
      singulars
      1. POconfused about httpclient performance
      1. USJames
    3. COyou are Network I/O bound, 100 threads are probably saturating your network connection. Then you are Disk I/O bound after that.
      singulars
      1. POconfused about httpclient performance
      1. USfeeling unwelcome
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload