Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>What you're looking for is a <a href="http://en.wikipedia.org/wiki/Sitemaps" rel="nofollow">sitemap</a>.</p> <p>First of all, you're writing a bot so it's good manners to check the blog's <a href="http://www.robotstxt.org/" rel="nofollow">robots.txt</a> file. And lo and behold, you'll often find a sitemap mentioned there. Here's an example from the <a href="http://googleblog.blogspot.ie/robots.txt" rel="nofollow">Google blog</a>:</p> <pre><code>User-agent: Mediapartners-Google Disallow: User-agent: * Disallow: /search Allow: / Sitemap: http://googleblog.blogspot.com/feeds/posts/default?orderby=UPDATED </code></pre> <p>In this case, you can visit the Sitemap URL to get an xml sitemap. </p> <p>For Wordpress, the same applies but it's not built-in as standard so not all blogs will have it. Have a look at <a href="http://wordpress.org/extend/plugins/google-sitemap-generator/" rel="nofollow">this plugin</a> which is the most popular way to create these sitemaps in Wordpress. For example, my blog uses this and you can find the sitemap at <a href="http://jamie.ideasasylum.com/sitemap.xml" rel="nofollow">/sitemap.xml</a> (the standard location)</p> <p>In short:</p> <ul> <li>Check robots.txt</li> <li>Follow the Sitemap url if it's present</li> <li>Otherwise, check for /sitemap.xml</li> </ul> <p><strong>Also:</strong> <a href="http://blog.mischel.com/2011/12/20/writing-a-web-crawler-politeness/" rel="nofollow">be a good Internet citizen</a>! If you're going to write a bot, make sure it obeys the robots.txt file (like where blogspot tells you explicitly <strong>not</strong> to use <code>/search</code>!)</p>
    singulars
    1. This table or related slice is empty.
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. This table or related slice is empty.
    2. VO
      singulars
      1. This table or related slice is empty.
    3. VO
      singulars
      1. This table or related slice is empty.
    1. This table or related slice is empty.
 

Querying!

 
Guidance

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload