Databases
StackOverflow2013
Postsdbo
POSimultaneous HTTP requests in PHP with cURL
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POSimultaneous HTTP requests in PHP with cURL
    primarykey
    data
    text
    <p>I'm trying to take a rather large list of domains query the rank of each using the compete.com API as seen here -> <a href="https://www.compete.com/developer/documentation" rel="nofollow">https://www.compete.com/developer/documentation</a></p> <p>The script I wrote takes a database of domains I populated and initiates a cURL request to compete for the rank of the website. I quickly realized that this was very slow because each request was being sent one at a time. I did some searching and came across this post-> <a href="http://www.phpied.com/simultaneuos-http-requests-in-php-with-curl/" rel="nofollow">http://www.phpied.com/simultaneuos-http-requests-in-php-with-curl/</a> which explains how to perform simultaneous HTTP requests in PHP with cURL. </p> <p>Unfortunately that script will take an array of 25,000 domains and try to process them all at once. I found that batches of 1,000 work quite well. </p> <p><em><strong>Any idea how to send 1,000 queries to compete.com then wait for completion and send the next 1,000 until the array is empty?</em></strong> Here's what I'm workin with thus far:</p> <pre><code>&lt;?php //includes include('includes/mysql.php'); include('includes/config.php'); //get domains $result = mysql_query("SELECT * FROM $tableName"); while($row = mysql_fetch_array($result)) { $competeRequests[] = "http://apps.compete.com/sites/" . $row['Domain'] . "/trended/rank/?apikey=xxx&amp;start_date=201207&amp;end_date=201208&amp;jsonp="; } //first batch $curlRequest = multiRequest($competeRequests); $j = 0; foreach ($curlRequest as $json){ $j++; $json_output = json_decode($json, TRUE); $rank = $json_output[data][trends][rank][0][value]; if($rank) { //Create mysql query $query = "Update $tableName SET Rank = '$rank' WHERE ID = '$j'"; //Execute the query mysql_query($query); echo $query . "&lt;br/&gt;"; } } function multiRequest($data) { // array of curl handles $curly = array(); // data to be returned $result = array(); // multi handle $mh = curl_multi_init(); // loop through $data and create curl handles // then add them to the multi-handle foreach ($data as $id =&gt; $d) { $curly[$id] = curl_init(); $url = (is_array($d) &amp;&amp; !empty($d['url'])) ? $d['url'] : $d; curl_setopt($curly[$id], CURLOPT_URL, $url); curl_setopt($curly[$id], CURLOPT_HEADER, 0); curl_setopt($curly[$id], CURLOPT_RETURNTRANSFER, 1); // post? if (is_array($d)) { if (!empty($d['post'])) { curl_setopt($curly[$id], CURLOPT_POST, 1); curl_setopt($curly[$id], CURLOPT_POSTFIELDS, $d['post']); } } curl_multi_add_handle($mh, $curly[$id]); } // execute the handles $running = null; do { curl_multi_exec($mh, $running); } while($running &gt; 0); // get content and remove handles foreach($curly as $id =&gt; $c) { $result[$id] = curl_multi_getcontent($c); curl_multi_remove_handle($mh, $c); } // all done curl_multi_close($mh); return $result; } ?&gt; </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USuser1647347
    1. USuser1647347
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POSimultaneous HTTP requests in PHP with cURL
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI didn't see a question in there.
      singulars
      1. POSimultaneous HTTP requests in PHP with cURL
      1. USFrank Farmer
    2. COLooking for a way to run 1000 domain batch rank check for 25k domains
      singulars
      1. POSimultaneous HTTP requests in PHP with cURL
      1. USuser1647347
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload