StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<blockquote> <p>Any idea on what kind of transformation we should do to that string to inflate it from Ruby?</p> </blockquote> <p><strong>UTF-8 to Latin-1</strong></p> <p>Ideally, there would be no need for any transformation, as long as you work with Buffers directly on the Node side. See pair of Node and Ruby code blocks at the very bottom below; however, the nature of the question is about what can be done on the Ruby side alone to address this. </p> <h2>Ruby-only - Convert from UTF-8 to LATIN-1</h2> <pre><code>require 'zlib' require 'rubygems' require 'redis' require 'iconv' redis = Redis.new def inflate(buffer) zstream = Zlib::Inflate.new buf = zstream.inflate(buffer) zstream.finish zstream.close buf end def convert(buffer) utf8_to_latin1 = Iconv.new("LATIN1//TRANSLIT//IGNORE", "UTF8") utf8_to_latin1.iconv(buffer) end value = redis.get("testkey") value = convert(value) puts inflate(value); </code></pre> <h3>Explanation</h3> <p>The above code uses iconv to convert the value retrieved from Redis from UTF-8 back to the intended bytes.</p> <p>When deflating in Node, the resulting buffer contains the correct zlib generated bytes; the result string from <code>toString('binary')</code>, character for character matches the contents of the deflate result buffer as well; however, by the time the deflate result is stored in Redis, it is UTF-8 encoded. An example: </p> <p>deflating the string "ABCABC" results in:</p> <pre><code><Buffer 78 9c 73 74 72 76 74 72 06 00 05 6c 01 8d> </code></pre> <p>Yet, Redis returns:</p> <pre><code><Buffer 78 c2 9c 73 74 72 76 74 72 06 00 05 6c 01 c2 8d> </code></pre> <p>Hypothesizing a bit, it would seem that the string resulting from <code>toString('binary')</code> ends up as argument to new Buffer(...) somewhere, perhaps in node-redis. In the absence of a specified encoding argument to new Buffer(), the default UTF-8 encoding is applied. (See first reference). Further hypothesizing, by using only buffers you avoid the need to create a buffer from the string, and as a result, avoid the UTF8 encoding, and so the correct deflate values make it in and out of Redis. </p> <p>References</p> <ul> <li><a href="http://nodejs.org/api/buffer.html#buffer_new_buffer_str_encoding" rel="nofollow">http://nodejs.org/api/buffer.html#buffer_new_buffer_str_encoding</a></li> <li><a href="http://blog.grayproductions.net/articles/encoding_conversion_with_iconv" rel="nofollow">http://blog.grayproductions.net/articles/encoding_conversion_with_iconv</a></li> <li><a href="http://ruby-doc.org/stdlib-1.8.7/libdoc/zlib/rdoc/Zlib/Inflate.html" rel="nofollow">http://ruby-doc.org/stdlib-1.8.7/libdoc/zlib/rdoc/Zlib/Inflate.html</a>.</li> </ul> <h2>Node</h2> <pre><code>var zlib = require('zlib'); var redis = require("redis").createClient(); var message = new Buffer('your stuff goes here.'); //var message = new Buffer(xml.toString()); redis.on("error", function (err) { console.log("Error " + err); }); redis.on("connect", function() { console.log(message); zlib.deflate(message, function(error, deflated) { console.log(deflated); redis.set("testkey",deflated,function (err, reply) { console.log(reply.toString()); }); }); }); </code></pre> <h2>Ruby</h2> <pre><code>require 'zlib' require 'rubygems' require 'redis' redis = Redis.new def inflate(buffer) zstream = Zlib::Inflate.new buf = zstream.inflate(buffer) zstream.finish zstream.close buf end value = redis.get("testkey") puts inflate(value) </code></pre>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload