Databases
StackOverflow2013
Postsdbo
POTrouble with UTF-8 CSV input in Python
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POTrouble with UTF-8 CSV input in Python
    primarykey
    data
    text
    <p>This seems like it should be an easy fix, but so far a solution has eluded me. I have a single column csv file with non-ascii chars saved in utf-8 that I want to read in and store in a list. I'm attempting to follow the principle of the <a href="http://nedbatchelder.com/text/unipain.html" rel="nofollow noreferrer">"Unicode Sandwich"</a> and decode upon reading the file in:</p> <pre><code>import codecs import csv with codecs.open('utf8file.csv', 'rU', encoding='utf-8') as file: input_file = csv.reader(file, delimiter=",", quotechar='|') list = [] for row in input_file: list.extend(row) </code></pre> <p>This produces the dread 'codec can't encode characters in position, ordinal not in range(128)' error.</p> <p>I've also tried adapting a solution from <a href="https://stackoverflow.com/questions/904041/reading-a-utf8-csv-file-with-python">this answer</a>, which returns a similar error</p> <pre><code>def unicode_csv_reader(utf8_data, dialect=csv.excel, **kwargs): csv_reader = csv.reader(utf8_data, dialect=dialect, **kwargs) for row in csv_reader: yield [unicode(cell, 'utf-8') for cell in row] filename = 'inputs\encode.csv' reader = unicode_csv_reader(open(filename)) target_list = [] for field1 in reader: target_list.extend(field1) </code></pre> <p>A very similar solution adapted from the <a href="http://docs.python.org/library/csv.html#examples" rel="nofollow noreferrer">docs</a> returns the same error.</p> <pre><code>def unicode_csv_reader(utf8_data, dialect=csv.excel): csv_reader = csv.reader(utf_8_encoder(utf8_data), dialect) for row in csv_reader: yield [unicode(cell, 'utf-8') for cell in row] def utf_8_encoder(unicode_csv_data): for line in unicode_csv_data: yield line.encode('utf-8') filename = 'inputs\encode.csv' reader = unicode_csv_reader(open(filename)) target_list = [] for field1 in reader: target_list.extend(field1) </code></pre> <p>Clearly I'm missing something. Most of the questions that I've seen regarding this problem seem to predate Python 2.7, so an update here might be useful.</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USCommunity
    1. USacpigeon
    plurals
    1. PL
      singulars
      1. LTLinked
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    2. PO
      singulars
      1. PTAnswer
    3. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. POTrouble with UTF-8 CSV input in Python
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. POTrouble with UTF-8 CSV input in Python
      1. This table or related slice is empty.
      1. VTUpMod
    3. VO
      singulars
      1. POTrouble with UTF-8 CSV input in Python
      1. This table or related slice is empty.
      1. VTUpMod
    1. COThe error message suggests it could be trying to decode the data as ASCII...
      singulars
      1. POTrouble with UTF-8 CSV input in Python
      1. USLev Levitsky
    2. COYour second example works for me, maybe you have broken csv module, I suggest you upgrade your python? Or maybe the error is elsewhere
      singulars
      1. POTrouble with UTF-8 CSV input in Python
      1. USAntti Haapala
    3. COIt sounds like your original CSV file isn't encoded as UTF-8 - can you confirm it is? Maybe it's UTF-16LE or something, or some other language-specific coding. You can use the chardet module to detect encoding.
      singulars
      1. POTrouble with UTF-8 CSV input in Python
      1. USAnsari
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload