Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>Certainly one thing you'll want to address is the <a href="http://en.wikipedia.org/wiki/Robots_exclusion_standard" rel="nofollow">Robots exclusion protocol</a> (aka <code>robots.txt</code>) which specifies which paths should be ignored by search engine crawlers; you should be able to update this per user preferences to prohibit future crawling.</p> <p>You might also look at Google's <a href="http://www.google.com/support/webmasters/bin/answer.py?hl=en&amp;answer=156184" rel="nofollow">Sitemaps</a> feature; this allows you to suggest to Google -- even for lists of pages that change a lot -- which URLs (in your case the URLs of your user profiles) Google should crawl. As they point out, this is no guarantee that every URL will be crawled, but it should help.</p> <p>What I don't have an answer for and which I'd be very curious to learn more about is some protocol for asking a search engine to remove URLs that it has already crawled. As far as I know, removing a page from robots.txt and sitemap.xml won't immediately make Google remove that page from its index, but will only slowly get pruned as the search engine decides it's out of date. Your users, on the other hand, will probably want their pages not to be searchable as soon as they check that particular preference. You may just need to educate users about this distinction, but I'd also be curious if their are programmatic ways to handle this. (For example, I believe Twitter has a way to ask Google to stop returning a tweet in search results after it's been deleted.)</p>
    singulars
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
 

Querying!

 
Guidance

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload