Databases
StackOverflow2013
Postsdbo
PODirect upload to S3 without CORS (non-browser)
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PODirect upload to S3 without CORS (non-browser)
    text
    copied!<p>We are developing an API on AWS, using a combination of EC2, SQS and S3. Our frontend servers reside on EC2, and we store all user-submitted data on S3. We don't want to expose our S3 storage to end users, so all submissions are initially pass through the EC2. All our end users are ever aware of is our API endpoints, to which they POST data (normally raw text, up to about 1MB at present), and from which they GET results.</p> <p>We suspect this model is unlikely to scale, however, as we acquire more users and as data submissions become larger. So we have been looking at alternative ways of managing user input. The most attractive of these seems to be <a href="http://en.wikipedia.org/wiki/Cross-origin_resource_sharing" rel="nofollow">CORS</a>, which <a href="http://docs.aws.amazon.com/AmazonS3/latest/dev/cors.html" rel="nofollow">S3 now supports</a>, and which allows end users to directly upload to S3 without having to initially buffer to EC2. But we are developing an API not a web application, and it seems as if CORS would always require some kind of browser-based Javascript to perform the actual upload. We'll probably have some kind of browser UI for large uploads eventually, but all we want to make available to developers right now is our API endpoints.</p> <p>So basically we are wondering if there is any other possible solutions we could look at. The most important thing for us at the moment is avoiding making public our backend storage, so that only our API urls are exposed (this is also important because we want to retain the possibility of swapping out our backend storage to something other than S3 in the future). We have also looked at the possibility of allowing users to submit using their own S3 accounts, which some APIs certainly do, but don't find this an unattractive option.</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload