Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>Yes, all the major search engines support basic wildcards <code>*</code>, and your solution will work to disallow your <strong>deals</strong> directory.</p> <p>The best place to learn about robots.txt is really the <a href="https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt" rel="nofollow">Google Developer page</a>. It provides plenty of examples of what works and what doesn't. For instance, many people don't know that robots.txt files are protocol specific. So if you want to block pages on a <code>https</code> connection, you'll need to make sure you have a robots.txt at <a href="https://yoursite.com/robots.txt" rel="nofollow">https://yoursite.com/robots.txt</a></p> <p>You can also test a new robots.txt file before applying it through <a href="http://www.google.com/webmasters/tools/" rel="nofollow">Google Webmaster Tools</a>. Basically you can verify with the search engine whether or not it'll actually work before you deploy it.</p> <p>With regards to blocking something with robots.txt or just adding a noindex to the pages, I'm more inclined to use the noindex in most scenarios unless I know I don't want the search engines crawling that section of my site at all.</p> <p>There are some trade offs. When you block the search engine altogether, you can save on some of your "crawl budget". So the search engines will crawl other pages then "waste" their time on pages you don't want them to visit. However, those URLs can still appear in the search results.</p> <p>If you absolutely don't want any search referral traffic to those pages, it's better to use the noindex directive. Additionally, if you link to the deals page often, a noindex not only removes it from the search results, but any link value / PageRank can flow through those pages and can be calculated accordingly. If you block them from being crawled, it's sort of a blackhole.</p>
    singulars
    1. This table or related slice is empty.
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. This table or related slice is empty.
    2. VO
      singulars
      1. This table or related slice is empty.
 

Querying!

 
Guidance

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload