Databases
StackOverflow2013
Postsdbo
POUpload a video file by chunks
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POUpload a video file by chunks
    primarykey
    data
    text
    <p><strong>Yes, it's a long question with a lot of detail...</strong> So, my question is: How can I stream an upload to Vimeo in segments?</p> <p><strong>For anyone wanting to copy and debug on their own machine:</strong> Here are the things you need:</p> <ul> <li>My code <a href="http://collabedit.com/8hwwy" rel="noreferrer">here</a>.</li> <li>Include the Scribe library found <a href="https://github.com/fernandezpablo85/scribe-java" rel="noreferrer">here</a></li> <li>Have a valid video file (mp4) which is at least greater than 10 MB and put it in the directory <code>C:\test.mp4</code> or change that code to point wherever yours is.</li> <li>That's it! Thanks for helping me out!</li> </ul> <p><strong>Big update:</strong> I've left a working API Key and Secret for Vimeo in the code <a href="http://collabedit.com/8hwwy" rel="noreferrer">here</a>. So as long as you have a Vimeo account, all the code should work just fine for you once you've allowed the application and entered your token. Just copy the code from that link into a project on your favorite IDE and see if you can fix this with me. I'll give the bounty to whoever gives me the working code. Thanks! Oh, and don't expect to use this Key and Secret for long. Once this problem's resolved I'll delete it. :)</p> <p><strong>Overview of the problem:</strong> The problem is when I send the last chunk of bytes to Vimeo and then verify the upload, the response returns that the length of all the content is the length of only the last chunk, not all the chunks combined as it should be.</p> <p><strong>SSCCE Note:</strong> I have my entire SSCCE <a href="http://collabedit.com/8hwwy" rel="noreferrer">here</a>. I put it somewhere else so it can be <strong>C</strong> ompilable. It is NOT very <strong>S</strong> hort (about 300 lines), but hopefully you find it to be <strong>S</strong> elf-contained, and it's certainly an <strong>E</strong> xample!). I am, however, posting the relevant portions of my code in this post.</p> <p><strong>This is how it works:</strong> When you upload a video to Vimeo via the streaming method (see Upload API documentation <a href="http://vimeo.com/api/docs/upload" rel="noreferrer">here</a> for setup to get to this point), you have to give a few headers: endpoint, content-length, and content-type. The documentation says it ignores any other headers. You also give it a payload of the byte information for the file you're uploading. And then sign and send it (I have a method which will do this using <a href="https://github.com/fernandezpablo85/scribe-java" rel="noreferrer">scribe</a>).</p> <p><strong>My problem:</strong> Everything works great when I just send the video in one request. My problem is in cases when I'm uploading several bigger files, the computer I'm using doesn't have enough memory to load all of that byte information and put it in the HTTP PUT request, so I have to split it up into 1 MB segments. This is where things get tricky. The documentation mentions that it's possible to "resume" uploads, so I'm trying to do that with my code, but it's not working quite right. Below, you'll see the code for sending the video. <strong>Remember</strong> my SSCCE is <a href="http://collabedit.com/8hwwy" rel="noreferrer">here</a>.</p> <p><strong>Things I've tried:</strong> I'm thinking it has something to do with the Content-Range header... So here are the things I've tried in changing what the Content-Range header says...</p> <ul> <li>Not adding content range header to the first chunk</li> <li><p>Adding a prefix to the content range header (each with a combination of the previous header):</p> <ul> <li>"bytes"</li> <li>"bytes " (throws connection error, see the very bottom for the error) --> It appears in the <a href="http://vimeo.com/api/docs/upload" rel="noreferrer">documentation</a> that this is what they're looking for, but I'm pretty sure there are typos in the documentation because they have the content-range header on their "resume" example as: <code>1001-339108/339108</code> when it should be <code>1001-339107/339108</code>. So... Yeah...</li> <li>"bytes%20"</li> <li>"bytes:"</li> <li>"bytes: "</li> <li>"bytes="</li> <li>"bytes= "</li> </ul></li> <li><p>Not adding anything as a prefix to the content range header</p></li> </ul> <p>Here's the code:</p> <pre><code>/** * Send the video data * * @return whether the video successfully sent */ private static boolean sendVideo(String endpoint, File file) throws FileNotFoundException, IOException { // Setup File long contentLength = file.length(); String contentLengthString = Long.toString(contentLength); FileInputStream is = new FileInputStream(file); int bufferSize = 10485760; // 10 MB = 10485760 bytes byte[] bytesPortion = new byte[bufferSize]; int byteNumber = 0; int maxAttempts = 1; while (is.read(bytesPortion, 0, bufferSize) != -1) { String contentRange = Integer.toString(byteNumber); long bytesLeft = contentLength - byteNumber; System.out.println(newline + newline + "Bytes Left: " + bytesLeft); if (bytesLeft &lt; bufferSize) { //copy the bytesPortion array into a smaller array containing only the remaining bytes bytesPortion = Arrays.copyOf(bytesPortion, (int) bytesLeft); //This just makes it so it doesn't throw an IndexOutOfBounds exception on the next while iteration. It shouldn't get past another iteration bufferSize = (int) bytesLeft; } byteNumber += bytesPortion.length; contentRange += "-" + (byteNumber - 1) + "/" + contentLengthString; int attempts = 0; boolean success = false; while (attempts &lt; maxAttempts &amp;&amp; !success) { int bytesOnServer = sendVideoBytes("Test video", endpoint, contentLengthString, "video/mp4", contentRange, bytesPortion, first); if (bytesOnServer == byteNumber) { success = true; } else { System.out.println(bytesOnServer + " != " + byteNumber); System.out.println("Success is not true!"); } attempts++; } first = true; if (!success) { return false; } } return true; } /** * Sends the given bytes to the given endpoint * * @return the last byte on the server (from verifyUpload(endpoint)) */ private static int sendVideoBytes(String videoTitle, String endpoint, String contentLength, String fileType, String contentRange, byte[] fileBytes, boolean addContentRange) throws FileNotFoundException, IOException { OAuthRequest request = new OAuthRequest(Verb.PUT, endpoint); request.addHeader("Content-Length", contentLength); request.addHeader("Content-Type", fileType); if (addContentRange) { request.addHeader("Content-Range", contentRangeHeaderPrefix + contentRange); } request.addPayload(fileBytes); Response response = signAndSendToVimeo(request, "sendVideo on " + videoTitle, false); if (response.getCode() != 200 &amp;&amp; !response.isSuccessful()) { return -1; } return verifyUpload(endpoint); } /** * Verifies the upload and returns whether it's successful * * @param endpoint to verify upload to * @return the last byte on the server */ public static int verifyUpload(String endpoint) { // Verify the upload OAuthRequest request = new OAuthRequest(Verb.PUT, endpoint); request.addHeader("Content-Length", "0"); request.addHeader("Content-Range", "bytes */*"); Response response = signAndSendToVimeo(request, "verifyUpload to " + endpoint, true); if (response.getCode() != 308 || !response.isSuccessful()) { return -1; } String range = response.getHeader("Range"); //range = "bytes=0-10485759" return Integer.parseInt(range.substring(range.lastIndexOf("-") + 1)) + 1; //The + 1 at the end is because Vimeo gives you 0-whatever byte where 0 = the first byte } </code></pre> <p>Here's the signAndSendToVimeo method:</p> <pre><code>/** * Signs the request and sends it. Returns the response. * * @param service * @param accessToken * @param request * @return response */ public static Response signAndSendToVimeo(OAuthRequest request, String description, boolean printBody) throws org.scribe.exceptions.OAuthException { System.out.println(newline + newline + "Signing " + description + " request:" + ((printBody &amp;&amp; !request.getBodyContents().isEmpty()) ? newline + "\tBody Contents:" + request.getBodyContents() : "") + ((!request.getHeaders().isEmpty()) ? newline + "\tHeaders: " + request.getHeaders() : "")); service.signRequest(accessToken, request); printRequest(request, description); Response response = request.send(); printResponse(response, description, printBody); return response; } </code></pre> <p>And here's <strong>some</strong> (an example... All of the output can be found <a href="http://collabedit.com/aqk8n" rel="noreferrer">here</a>) of the output from the printRequest and printResponse methods: <strong>NOTE</strong> This output changes depending on what the <code>contentRangeHeaderPrefix</code> is set to and the <code>first</code> boolean is set to (which specifies whether or not to include the Content-Range header on the first chunk).</p> <pre><code>We're sending the video for upload! Bytes Left: 15125120 Signing sendVideo on Test video request: Headers: {Content-Length=15125120, Content-Type=video/mp4, Content-Range=bytes%200-10485759/15125120} sendVideo on Test video &gt;&gt;&gt; Request Headers: {Authorization=OAuth oauth_signature="zUdkaaoJyvz%2Bt6zoMvAFvX0DRkc%3D", oauth_version="1.0", oauth_nonce="340477132", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="5cb447d1fc4c3308e2c6531e45bcadf1", oauth_token="460633205c55d3f1806bcab04174ae09", oauth_timestamp="1334336004", Content-Length=15125120, Content-Type=video/mp4, Content-Range=bytes: 0-10485759/15125120} Verb: PUT Complete URL: http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d sendVideo on Test video &gt;&gt;&gt; Response Code: 200 Headers: {null=HTTP/1.1 200 OK, Content-Length=0, Connection=close, Content-Type=text/plain, Server=Vimeo/1.0} Signing verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d request: Headers: {Content-Length=0, Content-Range=bytes */*} verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d &gt;&gt;&gt; Request Headers: {Authorization=OAuth oauth_signature="FQg8HJe84nrUTdyvMJGM37dpNpI%3D", oauth_version="1.0", oauth_nonce="298157825", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="5cb447d1fc4c3308e2c6531e45bcadf1", oauth_token="460633205c55d3f1806bcab04174ae09", oauth_timestamp="1334336015", Content-Length=0, Content-Range=bytes */*} Verb: PUT Complete URL: http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d &gt;&gt;&gt; Response Code: 308 Headers: {null=HTTP/1.1 308 Resume Incomplete, Range=bytes=0-10485759, Content-Length=0, Connection=close, Content-Type=text/plain, Server=Vimeo/1.0} Body: Bytes Left: 4639360 Signing sendVideo on Test video request: Headers: {Content-Length=15125120, Content-Type=video/mp4, Content-Range=bytes: 10485760-15125119/15125120} sendVideo on Test video &gt;&gt;&gt; Request Headers: {Authorization=OAuth oauth_signature="qspQBu42HVhQ7sDpzKGeu3%2Bn8tM%3D", oauth_version="1.0", oauth_nonce="183131870", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="5cb447d1fc4c3308e2c6531e45bcadf1", oauth_token="460633205c55d3f1806bcab04174ae09", oauth_timestamp="1334336015", Content-Length=15125120, Content-Type=video/mp4, Content-Range=bytes%2010485760-15125119/15125120} Verb: PUT Complete URL: http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d sendVideo on Test video &gt;&gt;&gt; Response Code: 200 Headers: {null=HTTP/1.1 200 OK, Content-Length=0, Connection=close, Content-Type=text/plain, Server=Vimeo/1.0} Signing verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d request: Headers: {Content-Length=0, Content-Range=bytes */*} verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d &gt;&gt;&gt; Request Headers: {Authorization=OAuth oauth_signature="IdhhhBryzCa5eYqSPKAQfnVFpIg%3D", oauth_version="1.0", oauth_nonce="442087608", oauth_signature_method="HMAC-SHA1", oauth_consumer_key="5cb447d1fc4c3308e2c6531e45bcadf1", oauth_token="460633205c55d3f1806bcab04174ae09", oauth_timestamp="1334336020", Content-Length=0, Content-Range=bytes */*} Verb: PUT Complete URL: http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d 4639359 != 15125120 verifyUpload to http://174.129.125.96:8080/upload?ticket_id=5ea64d64547e38e5e3c121852b2d306d &gt;&gt;&gt; Response Success is not true! Code: 308 Headers: {null=HTTP/1.1 308 Resume Incomplete, Range=bytes=0-4639359, Content-Length=0, Connection=close, Content-Type=text/plain, Server=Vimeo/1.0} Body: </code></pre> <p>Then the code goes on to complete the upload and set video information (you can see that in <a href="http://collabedit.com/8hwwy" rel="noreferrer">my full code</a>).</p> <p><strong>Edit 2:</strong> Tried removing the "%20" from the content-range and received this error making connection. I must use either "bytes%20" or not add "bytes" at all...</p> <pre><code>Exception in thread "main" org.scribe.exceptions.OAuthException: Problems while creating connection. at org.scribe.model.Request.send(Request.java:70) at org.scribe.model.OAuthRequest.send(OAuthRequest.java:12) at autouploadermodel.VimeoTest.signAndSendToVimeo(VimeoTest.java:282) at autouploadermodel.VimeoTest.sendVideoBytes(VimeoTest.java:130) at autouploadermodel.VimeoTest.sendVideo(VimeoTest.java:105) at autouploadermodel.VimeoTest.main(VimeoTest.java:62) Caused by: java.io.IOException: Error writing to server at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:622) at sun.net.www.protocol.http.HttpURLConnection.writeRequests(HttpURLConnection.java:634) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1317) at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468) at org.scribe.model.Response.&lt;init&gt;(Response.java:28) at org.scribe.model.Request.doSend(Request.java:110) at org.scribe.model.Request.send(Request.java:62) ... 5 more Java Result: 1 </code></pre> <p><strong>Edit 1:</strong> Updated the code and output. Still need help!</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USkentcdodds
    1. USkentcdodds
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POUpload a video file by chunks
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POUpload a video file by chunks
      1. USkentcdodds
      1. VTBountyStart
    3. VO
      singulars
      1. POUpload a video file by chunks
      1. This table or related slice is empty.
      1. VTUpMod
    1. COLooks like accounts other than yours are not able to use your API key + secret. However, I've found some more documentation on resumable uploads and found out that 1. the header should indeed look like "Content-Range: bytes 0-499/1000" and 2. the header for your last PUT should look like this "Content-Range: bytes 500-999/1000". http://code.google.com/apis/gdata/docs/resumable_upload.html#Resuming
      singulars
      1. POUpload a video file by chunks
      1. USTomTasche
    2. COWeird the Vimeo API thing isn't working. @anyone else: Is that the same for you? I can't seem to make my header look like you've indicated because it throws a connection exception when I use "bytes " as the prefix to the bytes I'm giving it (see error above).
      singulars
      1. POUpload a video file by chunks
      1. USkentcdodds
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload