Databases
StackOverflow2013
Postsdbo
POEncoding gives "'ascii' codec can't encode character … ordinal not in range(128)"
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POEncoding gives "'ascii' codec can't encode character … ordinal not in range(128)"
    text
    copied!<p>I am working through the Django RSS reader project <a href="http://code.djangoproject.com/browser/djangoproject.com/django_website/apps/aggregator/bin/update_feeds.py" rel="noreferrer">here</a>. </p> <p>The RSS feed will read something like "OKLAHOMA CITY (AP) — James Harden let". The RSS feed's encoding reads encoding="UTF-8" so I believe I am passing utf-8 to markdown in the code snippet below. The em dash is where it chokes. </p> <p>I get the Django error of "'ascii' codec can't encode character u'\u2014' in position 109: ordinal not in range(128)" which is an UnicodeEncodeError. In the variables being passed I see "OKLAHOMA CITY (AP) \u2014 James Harden". The code line that is not working is:</p> <pre><code>content = content.encode(parsed_feed.encoding, "xmlcharrefreplace") </code></pre> <p>I am using markdown 2.0, django 1.1, and python 2.4. </p> <p>What is the magic sequence of encoding and decoding that I need to do to make this work?</p> <hr> <p>(In response to Prometheus' request. I agree the formatting helps)</p> <p>So in views I add a smart_unicode line above the parsed_feed encoding line...</p> <pre><code>content = smart_unicode(content, encoding='utf-8', strings_only=False, errors='strict') content = content = content.encode(parsed_feed.encoding, "xmlcharrefreplace") </code></pre> <p>This pushes the problem to my models.py for me where I have </p> <pre><code>def save(self, force_insert=False, force_update=False): if self.excerpt: self.excerpt_html = markdown(self.excerpt) # super save after this </code></pre> <p>If I change the save method to have...</p> <pre><code>def save(self, force_insert=False, force_update=False): if self.excerpt: encoded_excerpt_html = (self.excerpt).encode('utf-8') self.excerpt_html = markdown(encoded_excerpt_html) </code></pre> <p>I get the error <strong>"'ascii' codec can't decode byte 0xe2 in position 141: ordinal not in range(128)"</strong> because now it reads "\xe2\x80\x94" where the em dash was</p>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload