Databases
StackOverflow2013
Postsdbo
POReading a UTF8 CSV file with Python
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POReading a UTF8 CSV file with Python
    text
    copied!<p>I am trying to read a CSV file with accented characters with Python (only French and/or Spanish characters). Based on the Python 2.5 documentation for the csvreader (<a href="http://docs.python.org/library/csv.html" rel="noreferrer">http://docs.python.org/library/csv.html</a>), I came up with the following code to read the CSV file since the csvreader supports only ASCII.</p> <pre><code>def unicode_csv_reader(unicode_csv_data, dialect=csv.excel, **kwargs): # csv.py doesn't do Unicode; encode temporarily as UTF-8: csv_reader = csv.reader(utf_8_encoder(unicode_csv_data), dialect=dialect, **kwargs) for row in csv_reader: # decode UTF-8 back to Unicode, cell by cell: yield [unicode(cell, 'utf-8') for cell in row] def utf_8_encoder(unicode_csv_data): for line in unicode_csv_data: yield line.encode('utf-8') filename = 'output.csv' reader = unicode_csv_reader(open(filename)) try: products = [] for field1, field2, field3 in reader: ... </code></pre> <p>Below is an extract of the CSV file I am trying to read:</p> <pre><code>0665000FS10120684,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Bleu 0665000FS10120689,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Gris 0665000FS10120687,SD1200IS,Appareil photo numérique PowerShot de 10 Mpx de Canon avec trépied (SD1200IS) - Vert ... </code></pre> <p>Even though I try to encode/decode to UTF-8, I am still getting the following exception:</p> <pre><code>Traceback (most recent call last): File ".\Test.py", line 53, in &lt;module&gt; for field1, field2, field3 in reader: File ".\Test.py", line 40, in unicode_csv_reader for row in csv_reader: File ".\Test.py", line 46, in utf_8_encoder yield line.encode('utf-8', 'ignore') UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 68: ordinal not in range(128) </code></pre> <p>How do I fix this?</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload