Databases
StackOverflow2013
Postsdbo
POPHP stalls on loop of curl function with bad url
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POPHP stalls on loop of curl function with bad url
    text
    copied!<p>I have a database of a few thousand URL's that I am checking for links on pages (end up looking for specific links) and so I am throwing the below function through a loop and every once and awhile one of the URL's is bad and then the entire program just stalls and stops running and starts building up memory used. I thought adding the CURLOPT_TIMEOUT would fix this but it didn't. Any ideas?</p> <pre><code>$options = array( CURLOPT_RETURNTRANSFER =&gt; true, // return web page CURLOPT_HEADER =&gt; false, // don't return headers CURLOPT_FOLLOWLOCATION =&gt; true, // follow redirects CURLOPT_ENCODING =&gt; "", // handle all encodings CURLOPT_USERAGENT =&gt; "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.13) Gecko/20080311 Firefox/2.0.0.13'", // who am i CURLOPT_AUTOREFERER =&gt; true, // set referer on redirect CURLOPT_TIMEOUT =&gt; 2, // timeout on response CURLOPT_MAXREDIRS =&gt; 10, // stop after 10 redirects CURLOPT_POST =&gt; 0, // i am sending post data CURLOPT_POSTFIELDS =&gt; $curl_data, // this are my post vars CURLOPT_SSL_VERIFYHOST =&gt; 0, // don't verify ssl CURLOPT_SSL_VERIFYPEER =&gt; false, // CURLOPT_VERBOSE =&gt; 1 // ); $ch = curl_init($url); curl_setopt_array($ch,$options); $content = curl_exec($ch); $err = curl_errno($ch); $errmsg = curl_error($ch) ; $header = curl_getinfo($ch); curl_close($ch); // $header['errno'] = $err; // $header['errmsg'] = $errmsg; $header['content'] = $content; #Extract the raw URl from the current one $scheme = parse_url($url, PHP_URL_SCHEME); //Ex: http $host = parse_url($url, PHP_URL_HOST); //Ex: www.google.com $raw_url = $scheme . '://' . $host; //Ex: http://www.google.com #Replace the relative link by an absolute one $relative = array(); $absolute = array(); #String to search $relative[0] = '/src="\//'; $relative[1] = '/href="\//'; #String to remplace by $absolute[0] = 'src="' . $raw_url . '/'; $absolute[1] = 'href="' . $raw_url . '/'; $source = preg_replace($relative, $absolute, $content); //Ex: src="/image/google.png" to src="http://www.google.com/image/google.png" return $source; </code></pre>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload