Databases
StackOverflow2013
Postsdbo
PONutch segments folder grows every day
 

Note that there are some explanatory texts on larger screens.

plurals
  1. PONutch segments folder grows every day
    primarykey
    data
    text
    <p>I have configured nutch/solr 1.6 to crawl/index every 12 hours an intranet with about 4000 documents and html pages.</p> <p>If I execute the crawler with an empty database the process takes about 30 minutes. When the crawling is executed for several days, it becomes very slow. Looking the log file it seems that this night the last step (SolrIndexer) started after 1 hour and 20 minutes and it took a bit more than 1 hour.</p> <p>Because the number of documents indexed doesn't grow, I'm wondering why it is so slow now.</p> <p>Nutch is executed with the following command:</p> <pre><code>bin/nutch crawl -urlDir urls -solr http://localhost:8983/solr -dir nutchdb -depth 15 -topN 3000 </code></pre> <p>The nutch-site.xml contains:</p> <pre><code>&lt;?xml version="1.0"?&gt; &lt;?xml-stylesheet type="text/xsl" href="configuration.xsl"?&gt; &lt;!-- Put site-specific property overrides in this file. --&gt; &lt;configuration&gt; &lt;property&gt; &lt;name&gt;http.agent.name&lt;/name&gt; &lt;value&gt;Internet Site Agent&lt;/value&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;plugin.includes&lt;/name&gt; &lt;value&gt;protocol-http|urlfilter-regex|parse-(tika|metatags)|index-(basic|anchor|metadata|more|http-header)|scoring-opic|urlnormalizer-(pass|regex|basic)&lt;/value&gt; &lt;/property&gt; &lt;!-- Used only if plugin parse-metatags is enabled. --&gt; &lt;property&gt; &lt;name&gt;metatags.names&lt;/name&gt; &lt;value&gt;description;keywords;published;modified&lt;/value&gt; &lt;description&gt; Names of the metatags to extract, separated by;. Use '*' to extract all metatags. Prefixes the names with 'metatag.' in the parse-metadata. For instance to index description and keywords, you need to activate the plugin index-metadata and set the value of the parameter 'index.parse.md' to 'metatag.description;metatag.keywords'. &lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;index.parse.md&lt;/name&gt; &lt;value&gt;metatag.description,metatag.keywords,metatag.published,metatag.modified&lt;/value&gt; &lt;description&gt; Comma-separated list of keys to be taken from the parse metadata to generate fields. Can be used e.g. for 'description' or 'keywords' provided that these values are generated by a parser (see parse-metatags plugin) &lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;db.ignore.external.links&lt;/name&gt; &lt;value&gt;true&lt;/value&gt; &lt;description&gt;Set this to false if you start crawling your website from for example http://www.example.com but you would like to crawl xyz.example.com. Set it to true otherwise if you want to exclude external links &lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;http.content.limit&lt;/name&gt; &lt;value&gt;10000000&lt;/value&gt; &lt;description&gt;The length limit for downloaded content using the http protocol, in bytes. If this value is nonnegative (&gt;=0), content longer than it will be truncated; otherwise, no truncation at all. Do not confuse this setting with the file.content.limit setting. &lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;fetcher.max.crawl.delay&lt;/name&gt; &lt;value&gt;1&lt;/value&gt; &lt;description&gt; If the Crawl-Delay in robots.txt is set to greater than this value (in seconds) then the fetcher will skip this page, generating an error report. If set to -1 the fetcher will never skip such pages and will wait the amount of time retrieved from robots.txt Crawl-Delay, however long that might be. &lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;fetcher.threads.fetch&lt;/name&gt; &lt;value&gt;10&lt;/value&gt; &lt;description&gt;The number of FetcherThreads the fetcher should use. This is also determines the maximum number of requests that are made at once (each FetcherThread handles one connection). The total number of threads running in distributed mode will be the number of fetcher threads * number of nodes as fetcher has one map task per node. &lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;fetcher.threads.fetch&lt;/name&gt; &lt;value&gt;10&lt;/value&gt; &lt;description&gt;The number of FetcherThreads the fetcher should use. This is also determines the maximum number of requests that are made at once (each FetcherThread handles one connection). The total number of threads running in distributed mode will be the number of fetcher threads * number of nodes as fetcher has one map task per node. &lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;fetcher.server.delay&lt;/name&gt; &lt;value&gt;1.0&lt;/value&gt; &lt;description&gt;The number of seconds the fetcher will delay between successive requests to the same server.&lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;http.redirect.max&lt;/name&gt; &lt;value&gt;0&lt;/value&gt; &lt;description&gt;The maximum number of redirects the fetcher will follow when trying to fetch a page. If set to negative or 0, fetcher won't immediately follow redirected URLs, instead it will record them for later fetching. &lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;fetcher.threads.per.queue&lt;/name&gt; &lt;value&gt;2&lt;/value&gt; &lt;description&gt;This number is the maximum number of threads that should be allowed to access a queue at one time. Replaces deprecated parameter 'fetcher.threads.per.host'. &lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;link.delete.gone&lt;/name&gt; &lt;value&gt;true&lt;/value&gt; &lt;description&gt;Whether to delete gone pages from the web graph.&lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;link.loops.depth&lt;/name&gt; &lt;value&gt;20&lt;/value&gt; &lt;description&gt;The depth for the loops algorithm.&lt;/description&gt; &lt;/property&gt; &lt;!-- moreindexingfilter plugin properties --&gt; &lt;property&gt; &lt;name&gt;moreIndexingFilter.indexMimeTypeParts&lt;/name&gt; &lt;value&gt;false&lt;/value&gt; &lt;description&gt;Determines whether the index-more plugin will split the mime-type in sub parts, this requires the type field to be multi valued. Set to true for backward compatibility. False will not split the mime-type. &lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;moreIndexingFilter.mapMimeTypes&lt;/name&gt; &lt;value&gt;false&lt;/value&gt; &lt;description&gt;Determines whether MIME-type mapping is enabled. It takes a plain text file with mapped MIME-types. With it the user can map both application/xhtml+xml and text/html to the same target MIME-type so it can be treated equally in an index. See conf/contenttype-mapping.txt. &lt;/description&gt; &lt;/property&gt; &lt;!-- Fetch Schedule Configuration --&gt; &lt;property&gt; &lt;name&gt;db.fetch.interval.default&lt;/name&gt; &lt;!-- for now always re-fetch everything --&gt; &lt;value&gt;10&lt;/value&gt; &lt;description&gt;The default number of seconds between re-fetches of a page (less than 1 day). &lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;db.fetch.interval.max&lt;/name&gt; &lt;!-- for now always re-fetch everything --&gt; &lt;value&gt;10&lt;/value&gt; &lt;description&gt;The maximum number of seconds between re-fetches of a page (less than one day). After this period every page in the db will be re-tried, no matter what is its status. &lt;/description&gt; &lt;/property&gt; &lt;!--property&gt; &lt;name&gt;db.fetch.schedule.class&lt;/name&gt; &lt;value&gt;org.apache.nutch.crawl.AdaptiveFetchSchedule&lt;/value&gt; &lt;description&gt;The implementation of fetch schedule. DefaultFetchSchedule simply adds the original fetchInterval to the last fetch time, regardless of page changes.&lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;db.fetch.schedule.adaptive.inc_rate&lt;/name&gt; &lt;value&gt;0.4&lt;/value&gt; &lt;description&gt;If a page is unmodified, its fetchInterval will be increased by this rate. This value should not exceed 0.5, otherwise the algorithm becomes unstable.&lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;db.fetch.schedule.adaptive.dec_rate&lt;/name&gt; &lt;value&gt;0.2&lt;/value&gt; &lt;description&gt;If a page is modified, its fetchInterval will be decreased by this rate. This value should not exceed 0.5, otherwise the algorithm becomes unstable.&lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;db.fetch.schedule.adaptive.min_interval&lt;/name&gt; &lt;value&gt;60.0&lt;/value&gt; &lt;description&gt;Minimum fetchInterval, in seconds.&lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;db.fetch.schedule.adaptive.max_interval&lt;/name&gt; &lt;value&gt;31536000.0&lt;/value&gt; &lt;description&gt;Maximum fetchInterval, in seconds (365 days). NOTE: this is limited by db.fetch.interval.max. Pages with fetchInterval larger than db.fetch.interval.max will be fetched anyway.&lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;db.fetch.schedule.adaptive.sync_delta&lt;/name&gt; &lt;value&gt;true&lt;/value&gt; &lt;description&gt;If true, try to synchronize with the time of page change. by shifting the next fetchTime by a fraction (sync_rate) of the difference between the last modification time, and the last fetch time.&lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;db.fetch.schedule.adaptive.sync_delta_rate&lt;/name&gt; &lt;value&gt;0.3&lt;/value&gt; &lt;description&gt;See sync_delta for description. This value should not exceed 0.5, otherwise the algorithm becomes unstable.&lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;db.fetch.schedule.adaptive.sync_delta_rate&lt;/name&gt; &lt;value&gt;0.3&lt;/value&gt; &lt;description&gt;See sync_delta for description. This value should not exceed 0.5, otherwise the algorithm becomes unstable.&lt;/description&gt; &lt;/property--&gt; &lt;property&gt; &lt;name&gt;fetcher.threads.fetch&lt;/name&gt; &lt;value&gt;1&lt;/value&gt; &lt;description&gt;The number of FetcherThreads the fetcher should use. This is also determines the maximum number of requests that are made at once (each FetcherThread handles one connection). The total number of threads running in distributed mode will be the number of fetcher threads * number of nodes as fetcher has one map task per node. &lt;/description&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;hadoop.tmp.dir&lt;/name&gt; &lt;value&gt;/opt/apache-nutch/tmp/&lt;/value&gt; &lt;/property&gt; &lt;!-- Boilerpipe --&gt; &lt;property&gt; &lt;name&gt;tika.boilerpipe&lt;/name&gt; &lt;value&gt;true&lt;/value&gt; &lt;/property&gt; &lt;property&gt; &lt;name&gt;tika.boilerpipe.extractor&lt;/name&gt; &lt;value&gt;ArticleExtractor&lt;/value&gt; &lt;/property&gt; &lt;/configuration&gt; </code></pre> <p>As you can see, I have configured nutch to always refetch all the documents. Because the site is small, it should be ok for now to refetch everything (the first time takes only 30 minutes...).</p> <p>I have noticed that in the folder crawldb/segments every day more or less 40 new segments are created. the disk size of the database of course is growing very fast.</p> <p>Is this the expected behaviour ? Is there something wrong with the configuration?</p>
    singulars
    1. PO
      singulars
      1. PTAnswer
    1. This table or related slice is empty.
    1. PTQuestion
    1. USMarco Altieri
    1. USMarco Altieri
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. PO
      singulars
      1. PTAnswer
    1. VO
      singulars
      1. PONutch segments folder grows every day
      1. This table or related slice is empty.
      1. VTUpMod
    1. COI have found the answear. It is necessary to manually delete all the segments that are older than the db.default.fetch.interval. In my case... all of them.
      singulars
      1. PONutch segments folder grows every day
      1. USMarco Altieri
    2. COPlease elaborate upon your answer and post it as an Answer. Then accept it as the correct answer. That'd help others looking for similar issues. If you found the answer via other sources (StackOverflow or otherwise) please post those links, and if necessary relevant excerpts.
      singulars
      1. PONutch segments folder grows every day
      1. USzEro
 

Querying!

 
Guidance
A row detail

Detail views are divided into sections. All the information in the data section comes from columns in the selected row. The other sections display data from other, related rows.

Related data can be related in a to-one or a to-many fashion. Captions of data related in a to-many fashion link to a list view showing a filtered view of the table.

Try moving around until you find a non-empty to-many entry and click on the label to get to one. You can move back to the root by clicking on the database name in the header.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload