StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
primarykey
Id
10356359
data
AcceptedAnswerId
0
AnswerCount
0
ClosedDate
CommentCount
0
CommunityOwnedDate
CreationDate
2012-04-27T18:58:39.450
FavoriteCount
0
LastActivityDate
2012-04-27T18:58:39.450
LastEditDate
LastEditorUserId
0
OwnerUserId
599075
ParentId
9395026
PostTypeId
2
Score
3
ViewCount
0
LastEditorDisplayName
text
Body
I am also having a problem with this, but I am using redis as a datastore. this is my crawler: <pre><code>require "rubygems" require "anemone" urls = File.open("urls.csv") opts = {discard_page_bodies: true, skip_query_strings: true, depth_limit:2000, read_timeout: 10} File.open("results.csv", "a") do |result_file| while row = urls.gets row_ = row.strip.split(',') if row_[1].start_with?("http://") url = row_[1] else url = "http://#{row_[1]}" end Anemone.crawl(url, options = opts) do |anemone| anemone.storage = Anemone::Storage.Redis puts "crawling #{url}" anemone.on_every_page do |page| next if page.body == nil if page.body.downcase.include?("sometext") puts "found one at #{url}" result_file.puts "#{row_[0]},#{row_[1]}" next end # end if end # end on_every_page end # end crawl end # end while # we're done puts "We're done." end # end File.open </code></pre> I applied the patch from <a href="https://github.com/wordtracker/anemone/commit/487601ba6e4012320da4983e38ba94011097e6c9" rel="nofollow">here</a> to my core.rb file in the anemone gem: <pre><code>35 # Prevent page_queue from using excessive RAM. Can indirectly limit ra te of crawling. You'll additionally want to use discard_page_bodies and/or a non-memory 'storage' option 36 :max_page_queue_size => 100, </code></pre> ... (The following used to be on line 155) <pre><code>157 page_queue = SizedQueue.new(@opts[:max_page_queue_size]) </code></pre> and I have an hourly cron job doing: <pre><code>#!/usr/bin/env python import redis r = redis.Redis() r.flushall() </code></pre> to try and keep redis' memory usage down. I'm restarting a giant crawl now, so we'll see how it goes! I'll report back with results...
Tags
Title
singulars
PostAcceptedAnswerId
1. This table or related slice is empty.
PostParentId
1. PORuby, Mongodb, Anemone: web crawler with possible memory leak?
 singulars
 PostTypePostTypeId
 PTQuestion
PostTypePostTypeId
1. PTAnswer
UserLastEditorUserId
1. This table or related slice is empty.
UserOwnerUserId
1. USAndbdrew
plurals
PostLinksPostIdRelatedPostId
1. This table or related slice is empty.
PostLinksRelatedPostIdPostId
1. This table or related slice is empty.
PostsAcceptedAnswerId
1. PORuby, Mongodb, Anemone: web crawler with possible memory leak?
 singulars
 PostTypePostTypeId
 PTQuestion
PostsParentIdCreationDate
1. This table or related slice is empty.
VotesPostIdCreationDate
1. VO
 singulars
 PostPostId
 PO
 UserUserId
 This table or related slice is empty.
 VoteTypeVoteTypeId
 VTAcceptedByOriginator
2. VO
 singulars
 PostPostId
 PO
 UserUserId
 This table or related slice is empty.
 VoteTypeVoteTypeId
 VTUpMod
CommentsPostId
1. This table or related slice is empty.

Querying!

Guidance

A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.