Databases
StackOverflow2013
Postsdbo
POnode.js uploading output of imagemagick to AWS S3
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POnode.js uploading output of imagemagick to AWS S3
    text
    copied!<p>I am having a problem manipulating the image with imagemagick then upload it to S3. The resulting object has a different (bigger) size and seems to be corrupt. If I make an intermediate step and save the output to local tmp file first and read it back, then upload the result everything seems fine. This the code that does NOT work.</p> <pre><code>im.resize({ srcData: imageObject.Body, width: variant.width, height: variant.height, customArgs: ['-auto-orient'] }, function(err, stdout, stderr) { if (err) { // This resize completed successfully log.err('Failed calling imageMagick, bail out', err); callback(err); return; } var fileName = cfg.aws.s3.uploadDir + photo.imageId + '/' + variant.width + 'x' + variant.height + '.jpg'; log.info('Storing image at S3 ' + fileName); //fs.writeFileSync('/tmp/xxx.jpg', stdout, 'binary'); //stdout = fs.readFileSync('/tmp/xxx.jpg'); var x = new Buffer(stdout); console.log(x); s3.putObject( { Bucket: cfg.aws.s3.bucket, Key: fileName, Body: x, ContentType: 'image/jpeg', ACL: 'public-read' }, function(err, data) { if (err) { // Failed saving to S3 log.error('Failed saving to S3', err); } callback(err); } ); }); </code></pre> <p>Uncomment the fileWriteSync and fileReadSync and it works properly.</p> <p>The output of the console.log(x) command in two cases: BAD:</p> <blockquote> <p>Buffer c3 bf c3 98 c3 bf c3 a0 00 10 4a 46 49 46 00 01 01 01 00 01 00 01 00 00 c3 bf c3 9b 00 43 00 06 04 05 06 05 04 06 06 05 06 07 07 06 08 0a 10 0a 0a 09 09 ...></p> </blockquote> <p>GOOD:</p> <blockquote> <p>Buffer ff d8 ff e0 00 10 4a 46 49 46 00 01 01 01 00 01 00 01 00 00 ff db 00 43 00 06 04 05 06 05 04 06 06 05 06 07 07 06 08 0a 10 0a 0a 09 09 0a 14 0e 0f 0c 10 ...></p> </blockquote> <p>as you can see good one is a proper jpeg, the bad one though contains similar sequences like 4a 46 49 46 = JFIF, however some bytes are off, and there are shifts, the whole file is bigger by about 20% in the bad case.</p> <p>Something to do with encoding? I've tried several things but I am lost at this point.</p> <p>Thanks!</p> <p><em><strong>Update #1:</em></strong> Apparently it is related to UTF encoding but I still don't completely understand what happens in this case. Apparently c3 bf c3 98 c3 bf c3 a0 00 10 4a 46 49 46 00 01 is UTF encoding of:</p> <pre><code>U+00FF LATIN SMALL LETTER Y WITH DIAERESIS character (&amp;#x00FF;) U+00D8 LATIN CAPITAL LETTER O WITH STROKE character (&amp;#x00D8;) U+00FF LATIN SMALL LETTER Y WITH DIAERESIS character (&amp;#x00FF;) U+00E0 LATIN SMALL LETTER A WITH GRAVE character (&amp;#x00E0;) U+0000 &lt;control&gt; character U+0010 &lt;control&gt; character U+004A LATIN CAPITAL LETTER J character U+0046 LATIN CAPITAL LETTER F character U+0049 LATIN CAPITAL LETTER I character U+0046 LATIN CAPITAL LETTER F character U+0000 &lt;control&gt; character U+0001 &lt;control&gt; character </code></pre> <p>when FF D8 FF .. is exactly what I was expecting.</p> <p>I know how to make the code work without temporary files (replace var <strong>x = new Buffer(stdout);</strong> with var <strong>x = new Buffer(stdout, 'binary')</strong> )</p> <p>However I still can't say I completely understand what happened here, this should without the Buffer() wrapping, which component has the problem? imagemagick? Buffer?</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload