Databases
StackOverflow2013
Postsdbo
POForward a file upload stream to S3 through Iteratee with Play2 / Scala
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POForward a file upload stream to S3 through Iteratee with Play2 / Scala
    primarykey
    data
    text
    <p>I've read some stuff about the possibility to send a file to S3 through Iteratee, which seems to permit to send so S3 chunks of a file as we receive them and avoid an OutOfMemory for large files for exemple.</p> <p>I've found this SO post which is probably almost what i need to do: <a href="https://stackoverflow.com/questions/11916911/play-2-x-reactive-file-upload-with-iteratees">Play 2.x : Reactive file upload with Iteratees</a> I don't really understand how to do it, or either if it's really available in Play 2.0.2 (because Sadek Brodi says foldM is available in Play 2.1 only for exemple)</p> <p>Can someone explain this in a simple way, for someone who have read some blog about Iteratees, and is not yet a Scala/Play2 expert?</p> <p>I don't even know if i should use a multipart body parser or something like that, but one thing i know is that i don't understand what this code is doing:</p> <pre><code>val consumeAMB = Traversable.takeUpTo[Array[Byte]](1028*1028) &amp;&gt;&gt; Iteratee.consume() val rechunkAdapter:Enumeratee[Array[Byte],Array[Byte]] = Enumeratee.grouped(consumeAMB) val writeToStore: Iteratee[Array[Byte],_] = Iteratee.foldM[Array[Byte],_](connectionHandle){ (c,bytes) =&gt; // write bytes and return next handle, probable in a Future } BodyParser( rh =&gt; (rechunkAdapter &amp;&gt;&gt; writeToStore).map(Right(_))) </code></pre> <hr> <p>By the way, what will be the difference in memory consumption compared to using classic Java InputStream / OutputStream. I actually am able to forward a 500mb file to S3 in a non-blocking way, with a very low memory consumption, without using Iteratees, using Java + AsyncHttpClient + Grizzly (but I guess it would also work with Netty).</p> <p>So what's the advantage of using Iteratee?</p> <p>One difference I can see is that the InputStream I get and forward to S3 is in my case backed by a temporary file (this is a CXF behavior), so it may not be as reactive as Play Iteratee</p> <p>But with Iteratees, if the Enumerator produces bytes received by the connection and forward them to S3 through an Iteratee, then if the connection to S3 is not good and the bytes can't be forwarded very fastly, where are stored the "pending" bytes?</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USSebastien Lorber
    plurals
    1. PL
      singulars
      1. LTLinked
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POForward a file upload stream to S3 through Iteratee with Play2 / Scala
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POForward a file upload stream to S3 through Iteratee with Play2 / Scala
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POForward a file upload stream to S3 through Iteratee with Play2 / Scala
      1. This table or related slice is empty.
      1. VTUpMod
    1. COA different way of file uploading would be to have your content directly uploaded to S3. In the ruby world there are some gems that can do that, but I'm not familiar with scala.
      singulars
      1. POForward a file upload stream to S3 through Iteratee with Play2 / Scala
      1. USThomas Klemm
    2. CODo you mean that we can directly make the browser upload to S3 without going through the backend server?
      singulars
      1. POForward a file upload stream to S3 through Iteratee with Play2 / Scala
      1. USSebastien Lorber
    3. COYes, that's right. A great ruby gem for that is [Carrierwave Direct](https://github.com/dwilkie/carrierwave_direct). The S3 Docs might hold something on how to build it from scratch [here](http://aws.amazon.com/articles/1434).
      singulars
      1. POForward a file upload stream to S3 through Iteratee with Play2 / Scala
      1. USThomas Klemm
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload