StackOverflow2013

Note that there are some explanatory texts on larger screens.

plurals

PO
text
Body
copied!<p>You can't control that in the robots.txt file. It's possible that some crawlers might support something like that, but none of the big ones do (as far as I know). </p> <p>Dynamically changing the robots.txt file is also a bad idea in a case like this. Most crawlers cache the robots.txt file for a certain time, and continue using it until they refresh the cache. If they cache it at the "right" time, they might crawl normally all day. If they cache it at the "wrong" time, they would stop crawling altogether (and perhaps even remove indexed URLs from their index). For instance, Google generally <a href="http://code.google.com/web/controlcrawlindex/docs/robots_txt.html" rel="nofollow">caches the robots.txt file for a day</a>, meaning that changes during the course of a day would not be visible to Googlebot. </p> <p>If crawling is causing too much load on your server, you can sometimes adjust the crawl rate for individual crawlers. For instance, for Googlebot you can do this in <a href="http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=48620" rel="nofollow">Google Webmaster Tools</a>. </p> <p>Additionally, when crawlers attempt to crawl during times of high load, you can always just serve them a <a href="http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html#sec10.5.4" rel="nofollow">503 HTTP result code</a>. This tells crawlers to check back at some later time (you can also specify a retry-after HTTP header if you know when they should come back). While I'd try to avoid doing this strictly on a time-of-day basis (this can block many other features, such as Sitemaps, contextual ads, or website verification and can slow down crawling in general), in exceptional cases it might make sense to do that. For the long run, I'd strongly recommend only doing this when your server load is really much too high to successfully return content to crawlers. </p>

Querying!

Guidance

An individual column

Larger individual text columns get their own page to allow for proper reading.

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload