Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>The answer is that in UTF-8, ASCII is just 1 byte, but that in general, most Western languages including English use a few characters here and there that require 2 bytes, so actual percentages vary. The Greek and Cyrillic languages all require at least 2 bytes per character in their script when encoded in UTF-8. </p> <p>Common Eastern languages require for their characters 3 bytes in UTF-8 but 2 in UTF-16. Note however that “uncommon” Eastern characters require 4 bytes in both UTF-8 and UTF-16 alike. </p> <p>3 is indeed only 50% greater than 2. But that is for a single code point only. It does not apply to an entire file. </p> <p>The actual percentage is impossible to state with precision, because you do not know whether the balance of code points down in the 1- or 2-byte UTF-8 range, or in the 4-byte UTF-8 range. If there is white space in the Asian text, then that is only byte of UTF-8, and yet it is a costly 2 bytes of UTF-16.</p> <p>These things do vary. You can only get precise numbers on precise text, not on general text. Code points in Asian text take 1, 2, 3, or 4 bytes of UTF-8, while in UTF-16 they variously require 2 or 4 bytes apiece. </p> <h1>Case Study</h1> <p>Compare the various languages’ Wikipedia pages on Tokyo to see what I mean. Even in Eastern languages, there is still plenty of ASCII going on. This alone makes your figures fluctuate. Consider:</p> <pre><code>Paras Lines Words Graphs Chars UTF16 UTF8 8:16 16:8 Language 519 1525 6300 43120 43147 86296 44023 51% 196% English 343 728 1202 8623 8650 17302 9173 53% 189% Welsh 541 1722 9013 57377 57404 114810 59345 52% 193% Spanish 529 1712 9690 63871 63898 127798 67016 52% 191% French 321 837 2442 18999 19026 38054 21148 56% 180% Hungarian 202 464 976 7140 7167 14336 11848 83% 121% Greek 348 937 2938 21439 21467 42936 36585 85% 117% Russian 355 788 613 6439 6466 12934 13754 106% 94% Chinese, simplified 209 419 243 2163 2190 4382 3331 76% 132% Chinese, traditional 461 1127 1030 25341 25368 50738 65636 129% 77% Japanese 410 925 2955 13942 13969 27940 29561 106% 95% Korean </code></pre> <p>Each of those is the Tokyo Wikipedia page <strong>saved as text,</strong> <em>not</em> as HTML. All text is in NFC, not in NFD. The meaning of each of the columns is as follows:</p> <ol> <li>Paras is the number of blankline separated text spans.</li> <li>Lines is the number of linebreak separated text spans.</li> <li>Words is the number of whitespace separated text spans.</li> <li>Graphs is the number of Unicode extended grapheme clusters, sometimes called glyphs. These are user-visible characters.</li> <li>Chars is the number of Unicode code points. These are, or should be, programmer-visible characters.</li> <li>UTF16 is how many bytes that takes up when the file is stored as UTF-16.</li> <li>UTF8 is how many bytes that takes up when the file is stored as UTF-8.</li> <li>8:16 is the ratio of UTF-8 size to UTF-16 size, expressed as a percentage.</li> <li>16:8 is the ratio of UTF-16 size to UTF-8 size, expressed as a percentage.</li> <li>Language is which version of the Tokyo page we’re talking about here.</li> </ol> <p>I’ve grouped the languages into Western Latin, Western non-Latin, and Eastern. Observations:</p> <ol> <li><p>Western languages that use the Latin script suffer terribly upon conversion from UTF-8 to UTF-16, with English suffering the most by expanding by 96% and Hungarian the least by expanding by 80%. All are huge.</p></li> <li><p>Western languages that do not use the Latin script still suffer, but only 15-20%.</p></li> <li><p><strong>Eastern languages <em>DO NOT SUFFER</em> in UTF-8 the way everyone claims that they do!</strong> Behold: </p> <ul> <li>In Korean and in (simplified) Chinese, you get only 6% bigger in UTF-8 than in UTF-16.</li> <li>In Japanese, you get only 29% bigger in UTF-8 than in UTF-16.</li> <li>The traditional Chinese actually <em>got smaller</em> in UTF-8 than in UTF-16! In fact, it costs 32% to use UTF-16 over UTF-8 for this sample. If you look at the Lines and Words columns, it looks that this might be due to white space usage. </li> </ul></li> </ol> <p>I hope that answers your question. There is simply no +50% to +100% size increase for Eastern languages when encoded in UTF-8 compared to when these same texts are encoded in UTF-16. Only when taking individual code points do you ever see numbers like that, which is a completely unreasonable metric.</p>
    singulars
    1. This table or related slice is empty.
    1. POat all times text encoded in UTF-8 will never give us more than a +50% file size of the same text encoded in UTF-16. true / false?
      singulars
      1. PTQuestion
    1. PTAnswer
    1. UStchrist
    1. UStchrist
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POat all times text encoded in UTF-8 will never give us more than a +50% file size of the same text encoded in UTF-16. true / false?
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    1. CObtw, is this statement right: take an input (any input) and encode it in UTF-8 and UTF-16. The UTF-8 file will NEVER be over 50% greater in file size compared to the UTF-16 file for ALL possible inputs. true / false?
      singulars
      1. PO
      1. USPacerier
    2. COYes, it is true. And rarely met. In the ASCII range, UTF-16 is 100% bigger than UTF-8. or 2:1. In the non-ASCII BMP range, UTF-8 is 50% bigger than UTF-16, which is 3:2 or 2:3 depending which way you look at it. In the non-BMP range, they are the same size, or 1:1.
      singulars
      1. PO
      1. UStchrist
    3. COWow, what a thorough answer! But are there really actively used asian languages that have 4-byte-characters?
      singulars
      1. PO
      1. USMichael Barth
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload