Databases
StackOverflow2013
Postsdbo
POOptimizing Bulk Uploads to Rackspace Cloud Files via PHP
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POOptimizing Bulk Uploads to Rackspace Cloud Files via PHP
    primarykey
    data
    text
    <p>We have an application that parses data from external sources and localizes it back, saving and resizing images as the final step of the process. Given the size of our processing [2 million images to date] we've been using Rackspace Files for hosting the data...</p> <pre><code>require('/var/libs/rackspace/cloudfiles.php'); $auth = new CF_Authentication('xxx', 'yyyy'); $auth-&gt;authenticate(); $conn = new CF_Connection($auth,true); $container = $conn-&gt;get_container('some container'); foreach ($lotsofitems as $onitem){ // check the record // save the image to disk with cURL // resize it into 4 more versions // post it to rackspace if(file_exists('/var/temp/'. $image_id . '_full'. $image_type)){ $object = $container-&gt;create_object($image_id . '_full' . $image_type); $object-&gt;load_from_filename('/var/temp/'. $image_id . '_full' . $image_type); unlink('/var/temp/'. $image_id . '_full' . $image_type); // remove the temp save } if(file_exists('/var/temp/'. $image_id . '_big'. $image_type)){ $object = $container-&gt;create_object($image_id . '_big' . $image_type); $object-&gt;load_from_filename('/var/temp/'. $image_id . '_big' . $image_type); unlink('/var/temp/'. $image_id . '_big' . $image_type); // remove the temp save } if(file_exists('/var/temp/'. $image_id . '_med'. $image_type)){ $object = $container-&gt;create_object($image_id . '_med' . $image_type); $object-&gt;load_from_filename('/var/temp/'. $image_id . '_med' . $image_type); unlink('/var/temp/'. $image_id . '_med' . $image_type); // remove the temp save } // delete the original // repeat } </code></pre> <p>After optimizing our parser, GD, etc, we've benchmarked the process and processing the image takes about 1 second but transferring the 5 image variations to Rackspace's is taking 2-5 seconds per item and at times spikes up to 10+. </p> <ul> <li>get image: 1341964436 </li> <li>got image: 1341964436 </li> <li>resized image: 1341964437</li> <li>clouded one image: 1341964446 </li> <li>clouded image: 1341964448</li> <li>finished with image: 1341964448</li> </ul> <p>Some additional points:</p> <ol> <li>Our processing servers are on Rackspace's cloud as well. </li> <li>There are 5 total image versions ranging from around 30kb to 2kb</li> <li>All images are saved to disk before the transfer and removed after</li> <li>Our containers [we use several overall but one per item] are CDN enabled</li> </ol> <p>Does anyone have suggestions with bulk transfers to Rackspace? Should we be reconnecting after a certain duration / number of requests? Optimizing our connection some other way? Or is it just about forking the processes and running a lot of calls.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. This table or related slice is empty.
    1. USTed S
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. COWhat size are the images you're uploading? Could be that... I doubt it though. Have you spoken to Rackspace Support? Might be worth logging a ticket or using the Live Chat.
      singulars
      1. POOptimizing Bulk Uploads to Rackspace Cloud Files via PHP
      1. USajtrichards
    2. COThe images are fairly small... under 50kb for the full size, a couple kb for the thumbs. Rackspace support had a few ideas but those are in place and we're still slower than we'd like...
      singulars
      1. POOptimizing Bulk Uploads to Rackspace Cloud Files via PHP
      1. USTed S
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload