StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

POHandling Indian Languages in BeautifulSoup
primarykey
Id
14412907
data
AcceptedAnswerId
14412961
AnswerCount
2
ClosedDate
CommentCount
0
CommunityOwnedDate
CreationDate
2013-01-19T09:24:33.290
FavoriteCount
0
LastActivityDate
2016-11-21T22:18:09.897
LastEditDate
LastEditorUserId
0
OwnerUserId
1765768
ParentId
0
PostTypeId
1
Score
3
ViewCount
810
LastEditorDisplayName
text
Body
I'm trying to scrape the <a href="http://en.wikipedia.org/wiki/NDTV" rel="nofollow">NDTV</a> website for news titles. <a href="http://archives.ndtv.com/articles/2012-01.html" rel="nofollow">This</a> is the page I'm using as a HTML source. I'm using BeautifulSoup (bs4) to handle the HTML code, and I've got everything working, except my code breaks when I encounter the hindi titles in the page I linked to. My code so far is : <pre><code>import urllib2 from bs4 import BeautifulSoup htmlUrl = "http://archives.ndtv.com/articles/2012-01.html" FileName = "NDTV_2012_01.txt" fptr = open(FileName, "w") fptr.seek(0) page = urllib2.urlopen(htmlUrl) soup = BeautifulSoup(page, from_encoding="UTF-8") li = soup.findAll( 'li') for link_tag in li: hypref = link_tag.find('a').contents[0] strhyp = str(hypref) fptr.write(strhyp) fptr.write("\n") </code></pre> The error I get is : <pre><code>Traceback (most recent call last): File "./ScrapeTemplate.py", line 30, in <module> strhyp = str(hypref) UnicodeEncodeError: 'ascii' codec can't encode characters in position 0-5: ordinal not in range(128) </code></pre> I got the same error even when I didn't include the <code>from_encoding</code> parameter. I initially used it as <code>fromEncoding</code>, but python warned me that it was deprecated usage. How do I fix this? From what I've read I need to either avoid the hindi titles or explicitly encode it into non-ascii text, but I don't know how to do that. Any help would be greatly appreciated!
Tags
<python><web-scraping><beautifulsoup>
Title
Handling Indian Languages in BeautifulSoup
singulars
PostAcceptedAnswerId
1. PO
 singulars
 PostTypePostTypeId
 PTAnswer
PostParentId
1. This table or related slice is empty.
PostTypePostTypeId
1. PTQuestion
UserLastEditorUserId
1. This table or related slice is empty.
UserOwnerUserId
1. USKitchi
plurals
PostLinksPostIdRelatedPostId
1. This table or related slice is empty.
PostLinksRelatedPostIdPostId
1. PL
 singulars
 LinkTypeLinkTypeId
 LTLinked
2. PL
 singulars
 LinkTypeLinkTypeId
 LTLinked
PostsAcceptedAnswerId
1. This table or related slice is empty.
PostsParentIdCreationDate
1. PO
 singulars
 PostTypePostTypeId
 PTAnswer
2. PO
 singulars
 PostTypePostTypeId
 PTAnswer
VotesPostIdCreationDate
1. VO
 singulars
 PostPostId
 POHandling Indian Languages in BeautifulSoup
 UserUserId
 This table or related slice is empty.
 VoteTypeVoteTypeId
 VTUpMod
2. VO
 singulars
 PostPostId
 POHandling Indian Languages in BeautifulSoup
 UserUserId
 This table or related slice is empty.
 VoteTypeVoteTypeId
 VTUpMod
3. VO
 singulars
 PostPostId
 POHandling Indian Languages in BeautifulSoup
 UserUserId
 This table or related slice is empty.
 VoteTypeVoteTypeId
 VTUpMod
CommentsPostId
1. This table or related slice is empty.

Querying!

Guidance

A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.