StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
primarykey
Id
9234800
data
AcceptedAnswerId
0
AnswerCount
0
ClosedDate
CommentCount
4
CommunityOwnedDate
CreationDate
2012-02-10T21:06:40.900
FavoriteCount
0
LastActivityDate
2015-03-13T23:51:53.763
LastEditDate
2015-03-13T23:51:53.763
LastEditorUserId
970175
OwnerUserId
598472
ParentId
9233356
PostTypeId
2
Score
6
ViewCount
0
LastEditorDisplayName
text
Body
When I hear someone uses curl_multi_exec it usually turns out they just load it with, say, 100 urls, then wait when all complete, and then process them all, and then start over with the next 100 urls... Blame me, I was doing so too, but then I found out that it is possible to remove/add handles to curl_multi while something is still in progress, And it really saves a lot of time, especially if you reuse already open connections. I wrote a small library to handle queue of requests with callbacks; I'm not posting full version here of course ("small" is still quite a bit of code), but here's a simplified version of the main thing to give you the general idea: <pre><code>public function launch() { $channels = $freeChannels = array_fill(0, $this->maxConnections, NULL); $activeJobs = array(); $running = 0; do { // pick jobs for free channels: while ( !(empty($freeChannels) || empty($this->jobQueue)) ) { // take free channel, (re)init curl handle and let // queued object set options $chId = key($freeChannels); if (empty($channels[$chId])) { $channels[$chId] = curl_init(); } $job = array_pop($this->jobQueue); $job->init($channels[$chId]); curl_multi_add_handle($this->master, $channels[$chId]); $activeJobs[$chId] = $job; unset($freeChannels[$chId]); } $pending = count($activeJobs); // launch them: if ($pending > 0) { while(($mrc = curl_multi_exec($this->master, $running)) == CURLM_CALL_MULTI_PERFORM); // poke it while it wants curl_multi_select($this->master); // wait for some activity, don't eat CPU while ($running < $pending && ($info = curl_multi_info_read($this->master))) { // some connection(s) finished, locate that job and run response handler: $pending--; $chId = array_search($info['handle'], $channels); $content = curl_multi_getcontent($channels[$chId]); curl_multi_remove_handle($this->master, $channels[$chId]); $freeChannels[$chId] = NULL; // free up this channel if ( !array_key_exists($chId, $activeJobs) ) { // impossible, but... continue; } $activeJobs[$chId]->onComplete($content); unset($activeJobs[$chId]); } } } while ( ($running > 0 && $mrc == CURLM_OK) || !empty($this->jobQueue) ); } </code></pre> In my version $jobs are actually of separate class, not instances of controllers or models. They just handle setting cURL options, parsing response and call a given callback onComplete. With this structure new requests will start as soon as something out of the pool finishes. Of course it doesn't really save you if not just retrieving takes time but processing as well... And it isn't a true parallel handling. But I still hope it helps. :) P.S. did a trick for me. :) Once 8-hour job now completes in 3-4 mintues using a pool of 50 connections. Can't describe that feeling. :) I didn't really expect it to work as planned, because with PHP it rarely works exactly as supposed... That was like "ok, hope it finishes in at least an hour... Wha... Wait... Already?! 8-O"
Tags
Title
singulars
PostAcceptedAnswerId
1. This table or related slice is empty.
PostParentId
1. PODownloading pages in parallel using PHP
 singulars
 PostTypePostTypeId
 PTQuestion
PostTypePostTypeId
1. PTAnswer
UserLastEditorUserId
1. USredolent
UserOwnerUserId
1. USSlava
plurals
PostLinksPostIdRelatedPostId
1. This table or related slice is empty.
PostLinksRelatedPostIdPostId
1. This table or related slice is empty.
PostsAcceptedAnswerId
1. PODownloading pages in parallel using PHP
 singulars
 PostTypePostTypeId
 PTQuestion
PostsParentIdCreationDate
1. This table or related slice is empty.
VotesPostIdCreationDate
1. VO
 singulars
 PostPostId
 PO
 UserUserId
 This table or related slice is empty.
 VoteTypeVoteTypeId
 VTUpMod
2. VO
 singulars
 PostPostId
 PO
 UserUserId
 This table or related slice is empty.
 VoteTypeVoteTypeId
 VTUpMod
3. VO
 singulars
 PostPostId
 PO
 UserUserId
 This table or related slice is empty.
 VoteTypeVoteTypeId
 VTUpMod
CommentsPostId

Querying!

Guidance

A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.