Databases
StackOverflow2013
Postsdbo
POcURL multi hanging/ignoring timeout
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POcURL multi hanging/ignoring timeout
    primarykey
    data
    text
    <p>I'm using a 'rolling' cURL multi implementation (like <a href="https://stackoverflow.com/questions/1387235/problem-with-curl-multi">this SO post</a>, based on <a href="http://code.google.com/p/rolling-curl/source/browse/trunk/RollingCurl.php" rel="nofollow noreferrer">this cURL code</a>). It works fine to process thousands of URLs using up to 100 requests at the same time, with 5 instances of the script running as daemons (yeah, I know, this should be written in C or something).</p> <p>Here's the problem: after processing ~200,000 urls (across the 5 instances) <code>curl_multi_exec()</code> seems to break for all instances of the script. I've tried shutting down the scripts, then restarting, and the same thing happens (not after 200,000 urls, but right on restart), <strong>the script hangs calling <code>curl_multi_exec()</code></strong>.</p> <p>I put the script into 'single' mode, processing one regular cURL handle at time, and that works fine (but it's not quite the speed I need). My logging leads me to suspect that it may have hit a patch of slow/problematic connections (since every so often it seems to process on URL then hang again), but <strong>that would mean my <code>CURLOPT_TIMEOUT</code> is being ignored for the individual handles</strong>. Or maybe it's just something with running that many requests through cURL. </p> <p><strong>Anyone heard of anything like this?</strong></p> <p>Sample code (again based on <a href="http://code.google.com/p/rolling-curl/source/browse/trunk/RollingCurl.php" rel="nofollow noreferrer">this</a>):</p> <pre><code>//some logging shows it hangs right here, only looping a time or two //so the hang seems to be in the curl call while(($execrun = curl_multi_exec($master, $running)) == CURLM_CALL_MULTI_PERFORM); //code to check for error or process whatever returned </code></pre> <p>I have <code>CURLOPT_TIMEOUT</code> set to <code>120</code>, but in the cases where <code>curl_multi_exec()</code> finally returns some data, it's after 10 minutes of waiting.</p> <p>I have a bunch of testing/checking yet to do, but thought maybe this might ring a bell with someone.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USTim Lytle
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POcURL multi hanging/ignoring timeout
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POcURL multi hanging/ignoring timeout
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POcURL multi hanging/ignoring timeout
      1. USAlexandr
      1. VTFavorite
    1. This table or related slice is empty.
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload