StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>I don't have a clear solution for PHP, but for Python I personally used <a href="http://pypi.python.org/pypi/chardet" rel="nofollow noreferrer">Universal Encoding Detector library</a> which does a pretty good job at guessing what encoding the file is being written as.</p> <p>Just to get you started, here's a Python script that I had used to do the conversion (the original purpose is that I wanted to converted a Japanese code base from a mixture of UTF-16 and Shift-JIS, which I made a default guess if chardet is not confident of detecting the encoding):</p> <pre><code>import sys import codecs import chardet from chardet.universaldetector import UniversalDetector """ Detects encoding Returns chardet result""" def DetectEncoding(fileHdl): detector = UniversalDetector() for line in fileHdl: detector.feed(line) if detector.done: break detector.close() return detector.result """ Reencode file to UTF-8 """ def ReencodeFileToUtf8(fileName, encoding): #TODO: This is dangerous ^^||, would need a backup option :) #NOTE: Use 'replace' option which tolerates errorneous characters data = codecs.open(fileName, 'rb', encoding, 'replace').read() open(fileName, 'wb').write(data.encode('utf-8', 'replace')) """ Main function """ if __name__=='__main__': # Check for arguments first if len(sys.argv) <> 2: sys.exit("Invalid arguments supplied") fileName = sys.argv[1] try: # Open file and detect encoding fileHdl = open(fileName, 'rb') encResult = DetectEncoding(fileHdl) fileHdl.close() # Was it an empty file? if encResult['confidence'] == 0 and encResult['encoding'] == None: sys.exit("Possible empty file") # Only attempt to reencode file if we are confident about the # encoding and if it's not UTF-8 encoding = encResult['encoding'].lower() if encResult['confidence'] >= 0.7: if encoding != 'utf-8': ReencodeFileToUtf8(fileName, encoding) else: # TODO: Probably you could make a default guess and try to encode, or # just simply make it fail except IOError: sys.exit('An IOError occured') </code></pre>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload