Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>The problem is not in your application. In fact, if you open Notepad directly, enter <code>0,1,0,1,0,1,0,1,</code> 20 times, save the file (ANSI encoding), and re-open the file, you will see the same behavior.</p> <p>By default, the text file will be written in UTF-8 encoding without a Byte Order Mark (BOM). When Notepad opens the file, it first must detect the proper encoding (e.g., Unicode or UTF8) based on only the contents of the text file. This is done based on statistical analysis, using the <a href="http://msdn.microsoft.com/en-us/library/windows/desktop/dd318672%28v=vs.85%29.aspx" rel="nofollow"><code>IsTextUnicode</code></a> API. The API notes that:</p> <blockquote> <p>The IS_TEXT_UNICODE_STATISTICS and IS_TEXT_UNICODE_REVERSE_STATISTICS tests use statistical analysis. These tests are not foolproof. The statistical tests assume certain amounts of variation between low and high bytes in a string, and some ASCII strings can slip through.</p> </blockquote> <p>In the example of <code>0,1,0,1,0,1,0,1</code> repeated 20 times, the <code>IsTextUnicode</code> function incorrectly indicated the text was Unicode-encoded rather than UTF-8 encoded. (This type false positive is perhaps most infamously present in <a href="http://en.wikipedia.org/wiki/Bush_hid_the_facts" rel="nofollow">this bug</a>.)</p> <p>As evidence, the following :</p> <pre><code>[DllImport("Advapi32", SetLastError = false)] static extern bool IsTextUnicode(byte[] buf, int len, ref int opt); ... int iter = 20; string test = string test = String.Join("", Enumerable.Repeat("0,1,0,1,0,1,0,1,", iter)); var bytes = UTF8Encoding.UTF8.GetBytes(test); int opt = 0x20; // IS_TEXT_UNICODE_STATISTICS; Console.WriteLine(IsTextUnicode(bytes, bytes.Length, ref opt)); </code></pre> <p>If <code>iter &gt; 10</code> (e.g., for more than 10 repetitions), the encoding will be interpreted, incorrectly, as Unicode.</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. This table or related slice is empty.
    2. VO
      singulars
      1. This table or related slice is empty.
    3. VO
      singulars
      1. This table or related slice is empty.
 

Querying!

 
Guidance

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload