Databases
StackOverflow2013
Postsdbo
PORequestTimeout uploading to S3 using PHP
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PORequestTimeout uploading to S3 using PHP
    primarykey
    data
    text
    <p>I am having trouble uploading files to S3 from on one of our servers. We use S3 to store our backups and all of our servers are running Ubuntu 8.04 with PHP 5.2.4 and libcurl 7.18.0. Whenever I try to upload a file Amazon returns a RequestTimeout error. I know there is a bug in our current version of libcurl preventing uploads of over 200MB. For that reason we split our backups into smaller files.</p> <p>We have servers hosted on Amazon's EC2 and servers hosted on customer's "private clouds" (a VMWare ESX box behind their company firewall). The specific server that I am having trouble with is hosted on a customer's private cloud.</p> <p>We use the Amazon S3 PHP Class from <a href="http://undesigned.org.za/2007/10/22/amazon-s3-php-class" rel="nofollow">http://undesigned.org.za/2007/10/22/amazon-s3-php-class</a>. I have tried 200MB, 100MB and 50MB files, all with the same results. We use the following to upload the files:</p> <pre><code>$s3 = new S3($access_key, $secret_key, false); $success = $s3-&gt;putObjectFile($local_path, $bucket_name, $remote_name, S3::ACL_PRIVATE); </code></pre> <p>I have tried setting <code>curl_setopt($curl, CURLOPT_NOPROGRESS, false);</code> to view the progress bar while it uploads the file. The first time I ran it with this option set it worked. However, every subsequent time it has failed. It seems to upload the file at around 3Mb/s for 5-10 seconds then drops to 0. After 20 seconds sitting at 0, Amazon returns the "RequestTimeout - Your socket connection to the server was not read from or written to within the timeout period. Idle connections will be closed." error.</p> <p>I have tried updating the S3 class to the latest version from <a href="https://github.com/tpyo/amazon-s3-php-class/blob/master/S3.php" rel="nofollow">GitHub</a> but it made no difference. I also found the <a href="http://www.phpclasses.org/package/4144-PHP-Stream-wrapper-to-get-and-send-files-to-Amazon-S3.html" rel="nofollow">Amazon S3 Stream Wrapper</a> class and gave that a try using the following code:</p> <pre><code>include 'gs3.php'; define('S3_KEY', 'ACCESSKEYGOESHERE'); define('S3_PRIVATE','SECRETKEYGOESHERE'); $local = fopen('/path/to/backup_id.tar.gz.0000', 'r'); $remote = fopen('s3://bucket-name/customer/backup_id.tar.gz.0000', 'w+r'); $count = 0; while (!feof($local)) { $result = fwrite($remote, fread($local, (1024 * 1024))); if ($result === false) { fwrite(STDOUT, $count++.': Unable to write!'."\n"); } else { fwrite(STDOUT, $count++.': Wrote '.$result.' bytes'."\n"); } } fclose($local); fclose($remote); </code></pre> <p>This code reads the file one MB at a time in order to stream it to S3. For a 50MB file, I get "1: Wrote 1048576 bytes" 49 times (the first number changes each time of course) but on the last iteration of the loop I get an error that says "Notice: fputs(): send of 8192 bytes failed with errno=11 Resource temporarily unavailable in /path/to/http.php on line 230".</p> <p>My first thought was that this is a networking issue. We called up the customer and explained the issue and asked them to take a look at their firewall to see if they were dropping anything. According to their network administrator the traffic is flowing just fine.</p> <p>I am at a loss as to what I can do next. I have been running the backups manually and using SCP to transfer them to another machine and upload them. This is obviously not ideal and any help would be greatly appreciated.</p> <p><strong>Update - 06/23/2011</strong></p> <p>I have tried many of the options below but they all provided the same result. I have found that even trying to scp a file from the server in question to another server stalls immediately and eventually times out. However, I can use scp to download that same file from another machine. This makes me even more convinced that this is a networking issue on the clients end, any further suggestions would be greatly appreciated.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USJamesArmes
    1. USJamesArmes
    plurals
    1. This table or related slice is empty.
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. PORequestTimeout uploading to S3 using PHP
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PORequestTimeout uploading to S3 using PHP
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PORequestTimeout uploading to S3 using PHP
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI've been in a similar situation with PHP, ended up acquiring a linux FTP command line wrapper/handler for PHP, perhaps a similar solution might work out for you. I have little or no faith in PHP5's stream network layer when it comes to doing heavy lifting like your situation.
      singulars
      1. PORequestTimeout uploading to S3 using PHP
      1. USDavid
    2. COYou should try isolating the problem to know which part of your configuration is failing you. Spin up another instance on the cloud using similar and different versions of PHP, etc. to see if you can duplicate the failure.
      singulars
      1. PORequestTimeout uploading to S3 using PHP
      1. USRob Bailey
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload