StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p><em>NOTE: this was written for Python 2.x. Not sure if applicable to 3.x.</em></p> <p>Your use of <code>str</code> for raw binary data in memory is correct.<br> [If you're using Python 2.6+, it's even better to use <code>bytes</code> which in 2.6+ is just an alias to <code>str</code> but expresses your intention better, and will help if one day you port the code to Python 3.]</p> <p>As others note, writing binary data through a codec is strange. A write codec <em>takes unicode</em> and <em>outputs bytes</em> into the file. You're trying to do it backwards, hence our confusion about your intentions...</p> <p>[And your diagnosis of the error looks correct: since the codec expects unicode, Python is decoding your str into unicode with the system's default encoding, which chokes.]</p> <p><strong>What you want to see in the output file?</strong></p> <ul> <li><p><strong>If the file should contain the binary data as-is</strong>:</p> <p>Then you must not send it through a codec; you must write it directly to the file. A codec encodes <em>everything</em> and can only emit valid encodings of unicode (in your case, valid UTF-8). There is no input you can give it to make it emit arbitrary byte sequences!</p> <ul> <li>If you require a <strong>mixture</strong> of UTF-8 and raw binary data, you should open the file directly, and intermix writes of <code>some_data</code> with <code>some_text.encode('utf8')</code>...</li> </ul> <p>Note however that mixing UTF-8 with raw arbitrary data is very bad design, because such files are very inconvenient to deal with! Tools that understand unicode will choke on the binary data, leaving you with not convenient way to even view (let alone modify) the file.</p></li> <li><p><strong>If you want a friendly representation of arbitrary bytes in unicode</strong>:</p> <p>Pass <code>data.encode('base64')</code> to the codec. Base64 produces only clean ascii (letters, numbers, and a little punctuation) so it can be clearly embedded in anything, it clearly looks to people as binary data, and it's reasonably compact (slightly over 33% overhead).</p> <p>P.S. you may note that <code>data.encode('base64')</code> is strange.</p> <ul> <li><p><code>.encode()</code> is supposed to take unicode but I'm giving it a string?! Python has several pseudo-codecs that convert str->str such as 'base64' and 'zlib'.</p></li> <li><p><code>.encode()</code> always returns an str but you'll feed it into a codec expecting unicode?! In this case it will only contain clean ascii, so it doesn't matter. You may write explicitly <code>data.encode('base64').encode('utf8')</code> if it makes you feel better.</p></li> </ul></li> <li><p><strong>If you need a 1:1 mapping from arbitrary bytes to unicode</strong>:</p> <p>Pass <code>data.decode('latin1')</code> to the codec. <code>latin1</code> maps bytes 0-255 to unicode characters 0-255, which is kinda elegant.</p> <p>The codec will, of course, encode your characters - 128-255 are encoded as 2 or 3 bytes in UTF-8 (surprisingly, the average overhead is 50%, more than base64!). This quite kills the "elegance" of having a 1:1 mapping.</p> <p>Note also that unicode characters 0-255 include nasty invisible/control characters (newline, formfeed, soft hyphen, etc.) making your binary data annoying to view in text editors.</p> <p>Considering these drawbacks, <strong>I do not recommend latin1</strong> unless you understand exactly why you want it.<br> I'm just mentioning it as the other "natural" encoding that springs to mind.</p></li> </ul>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload