Databases
StackOverflow2013
Postsdbo
POHandling variable special characters in python from pulled HTML pages
Body
 

Note that there are some explanatory texts on larger screens.

plurals
  1. POHandling variable special characters in python from pulled HTML pages
    text
    copied!<p>When I get html data using Mechanize, I store this into a variable, let's call it "HTML_RESPONSE". Once this is done, I then parse it and extract three things: Title, Short Description and Long Description.</p> <p>The problem I am facing is where the short or long description have the potential of containing characters such as - &amp;, £, $ and so forth.</p> <p>The problem arises when I try to put this into an XML and save it, since python freaks out when I try to decode these.</p> <p>For example here is a short description from the page:</p> <pre><code>S_DESC = "Senior VP of Treasury and Corporate Finance &amp; ERM, RTL Group, has been invited to the above conference to present a Case Study on Integrating Strategy and Risk into Enterprise Risk Management" </code></pre> <p>The way I am decoding - </p> <pre><code>#!/usr/bin/python # -*- coding: ISO-8859-1 -*- print S_DESC.decode('UTF-8').encode('ascii','xmlcharrefreplace') </code></pre> <p>This works fine on ampersands. If I then get a S_DESC with a pound sterling sign, my script breaks with this output:</p> <p><strong>UnicodeEncodeError: 'ascii' codec can't encode character u'\xa3'</strong></p> <p>Where this code fails in the portion of my script (the above exception is thrown in the very last line, every time I get a pound sterling sign). I would like to know if there is a universal way of telling python to just handle these chars on it's own. Making 100 functions for each possible incompatible character is not an option, in the same way, I am not prepared to sift through the whole website (2k + articles) in order to identify all special characters that "might" throw my code off....</p> <pre><code>XML = """ &lt;MAIN&gt; &lt;ITEM&gt; &lt;Author&gt;{0}&lt;/Author&gt; &lt;Author_UN&gt;{1}&lt;/Author_UN&gt; &lt;Date_Modified&gt;{2}&lt;/Date_Modified&gt; &lt;Date_Published&gt;{3}&lt;/Date_Published&gt; &lt;Default_Group_Rights&gt; {4} &lt;/Default_Group_Rights&gt; &lt;attachment&gt; &lt;file_name&gt;{5}&lt;/file_name&gt; &lt;file_extension&gt;{6}&lt;/file_extension&gt; &lt;file_stored_local&gt;{7}&lt;/file_stored_local&gt; &lt;/attachment&gt; &lt;title&gt;{8}&lt;/title&gt; &lt;sm_desc&gt;{9}&lt;/sm_desc&gt; &lt;lg_desc&gt; &lt;![CDATA[ {10} ]]&gt; &lt;/lg_desc&gt; &lt;/ITEM&gt; &lt;/MAIN&gt;""".format(author_soup, username, date_modified, published_date, xrights, attachment_text, file_extension, localstore, item_title.decode('UTF-8').encode('ascii','xmlcharrefreplace'), short_description.decode('UTF-8').encode('ascii','xmlcharrefreplace'), long_description.decode('UTF-8').encode('ascii','xmlcharrefreplace')) </code></pre> <h1>[EDIT]</h1> <p>This is a sample code I created which reflects the error perfectly, just in case someone wan't to have a swing at this?</p> <pre><code> #TESTING GROUND # -*- coding: UTF-8 -*- author_soup = "John Smith" username = "jsmith" date_modified = "25 December 2012, 15:42 PM" published_date = "25 December 2012, 15:42 PM" xrights = "r-w-x-x" attachment_text = "Random Attachment" file_extension = "txt" localstore = "../Local" item_title = "The NEw Financial Reforms of 2012" short_description = " £16 Billion Spent on new reforms backfire." long_description = '[&lt;p&gt;fullstory&lt;/p&gt;, &lt;p&gt;&lt;a class="external-link" href="http://business.timesonline.co.uk/tol/business/industry_sectors/banking_and_finance/article4526065.ece"&gt;http://business.timesonline.co.uk/tol/business/industry_sectors/banking_and_finance/article4526065.ece&lt;/a&gt;&lt;/p&gt;]' XML = """ &lt;MAIN&gt; &lt;ITEM&gt; &lt;Author&gt;{0}&lt;/Author&gt; &lt;Author_UN&gt;{1}&lt;/Author_UN&gt; &lt;Date_Modified&gt;{2}&lt;/Date_Modified&gt; &lt;Date_Published&gt;{3}&lt;/Date_Published&gt; &lt;Default_Group_Rights&gt; {4} &lt;/Default_Group_Rights&gt; &lt;attachment&gt; &lt;file_name&gt;{5}&lt;/file_name&gt; &lt;file_extension&gt;{6}&lt;/file_extension&gt; &lt;file_stored_local&gt;{7}&lt;/file_stored_local&gt; &lt;/attachment&gt; &lt;title&gt;{8}&lt;/title&gt; &lt;sm_desc&gt;{9}&lt;/sm_desc&gt; &lt;lg_desc&gt; &lt;![CDATA[ {10} ]]&gt; &lt;/lg_desc&gt; &lt;/ITEM&gt; &lt;/MAIN&gt;""".format(author_soup, username, date_modified, published_date, xrights, attachment_text, file_extension, localstore, item_title.decode('UTF-8'), short_description.decode('UTF-8'), long_description.decode('UTF-8')) </code></pre>
 

Querying!

 
Guidance
An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload