StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>Here is a function you might find useful: <code>utf8_to_latin9()</code>. It converts to <code>ISO-8859-15</code> (including EURO, which <code>ISO-8859-1</code> does not have), but also works correctly for the <code>UTF-8</code>-><code>ISO-8859-1</code> conversion part of a <code>ISO-8859-1</code>-><code>UTF-8</code>-><code>ISO-8859-1</code> round-trip.</p> <p>The function ignores invalid code points similar to <code>//IGNORE</code> flag for iconv, but does not recompose decomposed UTF-8 sequences; that is, it won't turn <code>U+006E U+0303</code> into <code>U+00F1</code>. I don't bother recomposing because iconv does not either.</p> <p>The function is very careful about the string access. It will never scan beyond the buffer. The output buffer must be one byte longer than length, because it always appends the end-of-string NUL byte. The function returns the number of characters (bytes) in output, not including the end-of-string NUL byte.</p> <pre><code>/* UTF-8 to ISO-8859-1/ISO-8859-15 mapper. * Return 0..255 for valid ISO-8859-15 code points, 256 otherwise. */ static inline unsigned int to_latin9(const unsigned int code) { /* Code points 0 to U+00FF are the same in both. */ if (code < 256U) return code; switch (code) { case 0x0152U: return 188U; /* U+0152 = 0xBC: OE ligature */ case 0x0153U: return 189U; /* U+0153 = 0xBD: oe ligature */ case 0x0160U: return 166U; /* U+0160 = 0xA6: S with caron */ case 0x0161U: return 168U; /* U+0161 = 0xA8: s with caron */ case 0x0178U: return 190U; /* U+0178 = 0xBE: Y with diaresis */ case 0x017DU: return 180U; /* U+017D = 0xB4: Z with caron */ case 0x017EU: return 184U; /* U+017E = 0xB8: z with caron */ case 0x20ACU: return 164U; /* U+20AC = 0xA4: Euro */ default: return 256U; } } /* Convert an UTF-8 string to ISO-8859-15. * All invalid sequences are ignored. * Note: output == input is allowed, * but input < output < input + length * is not. * Output has to have room for (length+1) chars, including the trailing NUL byte. */ size_t utf8_to_latin9(char *const output, const char *const input, const size_t length) { unsigned char *out = (unsigned char *)output; const unsigned char *in = (const unsigned char *)input; const unsigned char *const end = (const unsigned char *)input + length; unsigned int c; while (in < end) if (*in < 128) *(out++) = *(in++); /* Valid codepoint */ else if (*in < 192) in++; /* 10000000 .. 10111111 are invalid */ else if (*in < 224) { /* 110xxxxx 10xxxxxx */ if (in + 1 >= end) break; if ((in[1] & 192U) == 128U) { c = to_latin9( (((unsigned int)(in[0] & 0x1FU)) << 6U) | ((unsigned int)(in[1] & 0x3FU)) ); if (c < 256) *(out++) = c; } in += 2; } else if (*in < 240) { /* 1110xxxx 10xxxxxx 10xxxxxx */ if (in + 2 >= end) break; if ((in[1] & 192U) == 128U && (in[2] & 192U) == 128U) { c = to_latin9( (((unsigned int)(in[0] & 0x0FU)) << 12U) | (((unsigned int)(in[1] & 0x3FU)) << 6U) | ((unsigned int)(in[2] & 0x3FU)) ); if (c < 256) *(out++) = c; } in += 3; } else if (*in < 248) { /* 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx */ if (in + 3 >= end) break; if ((in[1] & 192U) == 128U && (in[2] & 192U) == 128U && (in[3] & 192U) == 128U) { c = to_latin9( (((unsigned int)(in[0] & 0x07U)) << 18U) | (((unsigned int)(in[1] & 0x3FU)) << 12U) | (((unsigned int)(in[2] & 0x3FU)) << 6U) | ((unsigned int)(in[3] & 0x3FU)) ); if (c < 256) *(out++) = c; } in += 4; } else if (*in < 252) { /* 111110xx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx */ if (in + 4 >= end) break; if ((in[1] & 192U) == 128U && (in[2] & 192U) == 128U && (in[3] & 192U) == 128U && (in[4] & 192U) == 128U) { c = to_latin9( (((unsigned int)(in[0] & 0x03U)) << 24U) | (((unsigned int)(in[1] & 0x3FU)) << 18U) | (((unsigned int)(in[2] & 0x3FU)) << 12U) | (((unsigned int)(in[3] & 0x3FU)) << 6U) | ((unsigned int)(in[4] & 0x3FU)) ); if (c < 256) *(out++) = c; } in += 5; } else if (*in < 254) { /* 1111110x 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx 10xxxxxx */ if (in + 5 >= end) break; if ((in[1] & 192U) == 128U && (in[2] & 192U) == 128U && (in[3] & 192U) == 128U && (in[4] & 192U) == 128U && (in[5] & 192U) == 128U) { c = to_latin9( (((unsigned int)(in[0] & 0x01U)) << 30U) | (((unsigned int)(in[1] & 0x3FU)) << 24U) | (((unsigned int)(in[2] & 0x3FU)) << 18U) | (((unsigned int)(in[3] & 0x3FU)) << 12U) | (((unsigned int)(in[4] & 0x3FU)) << 6U) | ((unsigned int)(in[5] & 0x3FU)) ); if (c < 256) *(out++) = c; } in += 6; } else in++; /* 11111110 and 11111111 are invalid */ /* Terminate the output string. */ *out = '\0'; return (size_t)(out - (unsigned char *)output); } </code></pre> <p>Note that you can add custom transliteration for specific code points in the <code>to_latin9()</code> function, but you are limited to one-character replacements.</p> <p>As it is currently written, the function can do in-place conversion safely: input and output pointers can be the same. The output string will never be longer than the input string. If your input string has room for an extra byte (for example, it has the NUL terminating the string), you can safely use the above function to convert it from UTF-8 to ISO-8859-1/15. I deliberately wrote it this way, because it should save you some effort in an embedded environment, although this approach is a bit limited wrt. customization and extension.</p> <p>Edit:</p> <p>I included a pair of conversion functions <a href="https://stackoverflow.com/questions/11258019/conversion-from-iso-8859-15-latin9-to-utf-8/11260683#11260683">in an edit to this answer</a> for both Latin-1/9 to/from UTF-8 conversion (ISO-8859-1 or -15 to/from UTF-8); the main difference is that those functions return a dynamically allocated copy, and keep the original string intact.</p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload