Databases
StackOverflow2013
Postsdbo
POHow use perl to process a file whose format is similar to unicode?
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHow use perl to process a file whose format is similar to unicode?
    primarykey
    data
    text
    <p>I have a legacy program, and after running it, it will generate a log file. Now I need to analysis this log file. </p> <p>But the file format is very strange. Please see the following,I used vi to open it, it looks like an unicode file, but it is not FFFE started. after I used notepad open it, save it and open again, I found that the FFFE is added by notepad. Then I can use command 'type log.txt > log1.txt" to convert the whole file to ANSI format. Later in perl, I can use /TDD/ in perl to search what I need. </p> <p>But now, I can't deal with this file format. </p> <p>Any comment or idea will be very appreciated. </p> <pre><code>0000000: 5400 4400 4400 3e00 2000 4c00 6f00 6100 T.D.D.&gt;. .L.o.a. </code></pre> <p>After notepad save it</p> <pre><code>0000000: fffe 5400 4400 4400 3e00 2000 4c00 6f00 ..T.D.D.&gt;. .L.o. open STDIN, "&lt; log.txt"; while(&lt;&gt;) { if (/TDD/) { # Add my logic. } } </code></pre> <p>I have read the thread which is very useful, but still can't resolve my problem. <a href="https://stackoverflow.com/questions/2461737/how-can-i-open-a-unicode-file-with-perl">How can I open a Unicode file with Perl?</a></p> <p>I can't add answer, so I edit my thread. </p> <p>Thanks Michael, I tried your script but got the following error. I checked my perl version is 5.1, OS is windows 2008. </p> <pre><code>* ascii * ascii-ctrl * iso-8859-1 * null * utf-8-strict * utf8 UTF-16:Unrecognised BOM 5400 at test.pl line 12. </code></pre> <p><strong>Update</strong></p> <p>I tried the UTF-16LE with the command: </p> <pre><code>perl.exe open.pl utf-16le utf-16 &lt;my log file&gt;.txt </code></pre> <p>but I still got the error like </p> <pre><code>UTF-16LE:Partial character at open.pl line 18, &lt;$fh&gt; line 1824. </code></pre> <p>also, I tried utf-16be, got the same error. </p> <p>If I used utf-16, I will got the error </p> <pre><code>UTF-16:Unrecognised BOM 5400 at open.pl line 18. </code></pre> <p>open.pl line 18 </p> <pre><code>is "print while &lt;$fh&gt;;" </code></pre> <p>Any idea? </p> <p>Updated: 5/11/2011. Thank you guys for your help. I resolved the problem. I found that the data in log file are not UTF-16 after all. So, I had to write a .net project by visual studio. It will read the log file with UTF-16 and write to a new file with UTF-8. And then I used perl script to parse the file and generate result data. It worked now. </p> <p>So, if any of you know how to use perl read a file with many garbage data, please tell me, thank you very much. </p> <p>e.g. garbage data sample</p> <pre><code>tests.cpp:34) ਍吀䐀䐀㸀 䰀漀愀搀椀渀最 挀挀洀挀漀爀攀⸀搀氀 </code></pre> <p>use hex reader to open it: </p> <pre><code>0000070: a88d e590 80e4 9080 e490 80e3 b880 e280 ................ 0000080: 80e4 b080 e6bc 80e6 8480 e690 80e6 a480 ................ 0000090: e6b8 80e6 9c80 e280 80e6 8c80 e68c 80e6 ................ 00000a0: b480 e68c 80e6 bc80 e788 80e6 9480 e2b8 ................ </code></pre>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USOrionpax
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POHow use perl to process a file whose format is similar to unicode?
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POHow use perl to process a file whose format is similar to unicode?
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POHow use perl to process a file whose format is similar to unicode?
      1. This table or related slice is empty.
      1. VTApproveEditSuggestion
    1. COMay this be of some help: http://www.perlmonks.org/?node_id=615796
      singulars
      1. POHow use perl to process a file whose format is similar to unicode?
      1. USAndrey Adamovich
    2. COHave you tried specifying different encodings for stream reading?
      singulars
      1. POHow use perl to process a file whose format is similar to unicode?
      1. USAndrey Adamovich
    3. COThe accepted answer of the question you linked to tells you exactly what you need to do to read your UCS-2LE encoded file. I just tried it and it worked for me. - Be more specific for what reason you "can't resolve".
      singulars
      1. POHow use perl to process a file whose format is similar to unicode?
      1. USdaxim
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload