Databases
StackOverflow2013
Postsdbo
PO
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>The file's meta tag claims that the character set is <a href="http://en.wikipedia.org/wiki/GB_2312" rel="nofollow">GB2312</a>, but the data contains a character from the newer <a href="http://en.wikipedia.org/wiki/GBK" rel="nofollow">GBK</a>/<a href="http://en.wikipedia.org/wiki/GB18030" rel="nofollow">GB18030</a> and this is what's tripping BeautifulSoup up:</p> <pre> simon@lucifer:~$ python Python 2.7 (r27:82508, Jul 3 2010, 21:12:11) [GCC 4.0.1 (Apple Inc. build 5493)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> import urllib2 >>> data = urllib2.urlopen('http://stock.eastmoney.com/news/1408,20101022101395594.html').read() >>> data.decode("gb2312") Traceback (most recent call last): File "", line 1, in UnicodeDecodeError: 'gb2312' codec can't decode bytes in position 20148-20149: illegal multibyte sequence </pre> <p>At this point, UnicodeDammit bails out, tries <a href="http://chardet.feedparser.org/" rel="nofollow">chardet</a>, <a href="http://en.wikipedia.org/wiki/UTF-8" rel="nofollow">UTF-8</a> and finally <a href="http://en.wikipedia.org/wiki/Windows-1252" rel="nofollow">Windows-1252</a>, which always succeeds - this is what you got, by the looks of it.</p> <p>If we tell the decoder to replace unrecognised characters with a '?', we can see the character that's missing in GB2312:</p> <pre> >>> print data[20140:20160].decode("gb2312", "replace") 毒尾气二�英的排放难 </pre> <p>Using the correct encoding:</p> <pre> >>> print data[20140:20160].decode("gb18030", "replace") 毒尾气二噁英的排放难 >>> from BeautifulSoup import BeautifulSoup >>> s = BeautifulSoup(data, fromEncoding="gb18030") >>> print s.findAll("p")[2].string[:10] 　　信息通信技术是&amp; </pre> <p>Also:</p> <pre> >>> print s.findAll("p")[2].string 　　信息通信技术是&amp;ldquo;十二五&amp;rdquo;规划重点发展方向，行业具有很强的内在增长潜 力，增速远高于GDP。软件外包、服务外包、管理软件、车载导航、网上购物、网络游戏、 移动办公、移动网络游戏、网络视频等均存在很强的潜在需求，使信息技术行业继续保持较 高增长。 </pre>
    singulars
    1. This table or related slice is empty.
    1. POPython - BeautifulSoup html parsing handle gbk encoding poorly - Chinese webscraping problem
      singulars
      1. PTQuestion
    1. PTAnswer
    1. USSimonJ
    1. USSimonJ
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. POPython - BeautifulSoup html parsing handle gbk encoding poorly - Chinese webscraping problem
      singulars
      1. PTQuestion
    1. This table or related slice is empty.
    1. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTUpMod
    2. VO
      singulars
      1. PO
      1. This table or related slice is empty.
      1. VTAcceptedByOriginator
    1. CO+1. Browsers have long used the Windows code pages (in this code page 936, which is essentially GBK) instead of the official-standard encodings specified by web pages like GB2312. This behaviour is finally being documented and standardised by HTML5 (section 8.2.2.2). It would be a good idea for BeautifulSoup to add this mapping.
      singulars
      1. PO
      1. USbobince
    2. COThis *almost* works perfectly, thanks so much for putting me on the right track! The only problem is now some of the punctuation marks get messed up. u'\u3002' displays funny. I think it's just a display problem as I checked the unicode site and confirmed that this is the full stop, but it displays in the upper right corner of the character space and not the lower left corner. Same with \u3001, the chinese serial comma. Any thoughts?
      singulars
      1. PO
      1. USWheaton
    3. COAlso, just to add this to posterity - if you try to print the p[2].string from, for example, the <p> tags above, you get a "maximum recursion" error. You can solve this by printing p[2].string.split() Figured that out on by accident, but it's really helpful
      singulars
      1. PO
      1. USWheaton
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload