StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
primarykey
Id
421335
data
AcceptedAnswerId
0
AnswerCount
0
ClosedDate
CommentCount
0
CommunityOwnedDate
CreationDate
2009-01-07T18:03:35.447
FavoriteCount
0
LastActivityDate
2009-01-07T18:08:42.037
LastEditDate
2009-01-07T18:08:42.037
LastEditorUserId
17160
OwnerUserId
17160
ParentId
421283
PostTypeId
2
Score
7
ViewCount
0
LastEditorDisplayName
nosklo
text
Body
Use <a href="http://scrapy.org/" rel="noreferrer">Scrapy</a>. It is a twisted-based web crawler framework. Still under heavy development but it works already. Has many goodies: <ul> <li>Built-in support for parsing HTML, XML, CSV, and Javascript</li> <li>A media pipeline for scraping items with images (or any other media) and download the image files as well</li> <li>Support for extending Scrapy by plugging your own functionality using middlewares, extensions, and pipelines</li> <li>Wide range of built-in middlewares and extensions for handling of compression, cache, cookies, authentication, user-agent spoofing, robots.txt handling, statistics, crawl depth restriction, etc</li> <li>Interactive scraping shell console, very useful for developing and debugging</li> <li>Web management console for monitoring and controlling your bot</li> <li>Telnet console for low-level access to the Scrapy process</li> </ul> Example code to extract information about all torrent files added today in the <a href="http://www.mininova.org/" rel="noreferrer">mininova</a> torrent site, by using a XPath selector on the HTML returned: <pre><code>class Torrent(ScrapedItem): pass class MininovaSpider(CrawlSpider): domain_name = 'mininova.org' start_urls = ['http://www.mininova.org/today'] rules = [Rule(RegexLinkExtractor(allow=['/tor/\d+']), 'parse_torrent')] def parse_torrent(self, response): x = HtmlXPathSelector(response) torrent = Torrent() torrent.url = response.url torrent.name = x.x("//h1/text()").extract() torrent.description = x.x("//div[@id='description']").extract() torrent.size = x.x("//div[@id='info-left']/p[2]/text()[2]").extract() return [torrent] </code></pre>
Tags
Title
singulars
PostAcceptedAnswerId
1. This table or related slice is empty.
PostParentId
1. POWhat are the best prebuilt libraries for doing Web Crawling in Python
 singulars
 PostTypePostTypeId
 PTQuestion
PostTypePostTypeId
1. PTAnswer
UserLastEditorUserId
1. USnosklo
UserOwnerUserId
1. USnosklo
plurals
PostLinksPostIdRelatedPostId
1. This table or related slice is empty.
PostLinksRelatedPostIdPostId
1. This table or related slice is empty.
PostsAcceptedAnswerId
1. POWhat are the best prebuilt libraries for doing Web Crawling in Python
 singulars
 PostTypePostTypeId
 PTQuestion
PostsParentIdCreationDate
1. This table or related slice is empty.
VotesPostIdCreationDate
1. VO
 singulars
 PostPostId
 PO
 UserUserId
 This table or related slice is empty.
 VoteTypeVoteTypeId
 VTUpMod
2. VO
 singulars
 PostPostId
 PO
 UserUserId
 This table or related slice is empty.
 VoteTypeVoteTypeId
 VTUpMod
3. VO
 singulars
 PostPostId
 PO
 UserUserId
 This table or related slice is empty.
 VoteTypeVoteTypeId
 VTUpMod
CommentsPostId
1. This table or related slice is empty.

Querying!

Guidance

A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.