Note that there are some explanatory texts on larger screens.

plurals
  1. PO
    primarykey
    data
    text
    <p>Don't know if this helps or not. But I promised to maintain an article on robots.txt for WordPres / Joomla / PHPBB. It's still in working condition. Here it is:</p> <p><a href="http://mast3rpee.tk/?p=127" rel="nofollow">http://mast3rpee.tk/?p=127</a></p> <p>Basically what I'm doing is setting a default robot.txt and also modifying apache using .haccess file to force loading of a custom robots.php. Why? Well this solves most problems not just the one here. It also works on free hosting which can create all sorts of issues (some ban txt files! some overide robots.txt etc).</p> <p>HERE IS THE CODE</p> <h2>.htaccess</h2> <pre><code># BEGIN Robots &lt;IfModule mod_rewrite.c&gt; &lt;FilesMatch "^robots.(txt|php)$"&gt; Header Set Last-Modified "Tue, 01 Jan 2013 12:00:00 GMT" &lt;/FilesMatch&gt; RewriteEngine On RewriteBase / RewriteRule ^(robots)\.txt$ /$1.php [L] &lt;/IfModule&gt; # END Robots </code></pre> <h2>robots.txt</h2> <pre><code>User-agent: * Disallow: /cgi-bin/ Disallow: /feed/ Disallow: /wp-admin/ Disallow: /wp-content/plugins/ Sitemap: http://{PUT YOUR DOMAIN}/sitemap.xml Crawl-delay: 4 </code></pre> <h2>robots.php</h2> <pre><code>&lt;?php $start = "2013/01/01"; // Date you started your blog YYYY/MM/DD $average = 30; // Number of posts you make per month $sitemap = "http://{PUT YOUR DOMAIN}/sitemap.xml"; // Is blog old enough $old = ($average/30)*(time()-strtotime($start)) &gt; 3600*24*360? true : false; // Output proper headers header ('Content-Type: text/plain'); header ('Cache-Control: private, pre-check=0, post-check=0, max-age=36000'); header ('Expires: ' . gmstrftime('%a, %d %b %Y %H:%M:%S GMT', time() + 36000)); header ('Last-Modified: ' . gmstrftime('%a, %d %b %Y %H:%M:%S GMT', time() - 36000)); if ($old) { $custom = date("Y/m"); echo &lt;&lt;&lt; ROBOTS User-agent: * Disallow: /cgi-bin/ Disallow: /feed/ Disallow: /wp-admin/ Disallow: /wp-content/plugins/ Disallow: /$custom Sitemap: $sitemap Crawl-delay: 4 ROBOTS; } else { $custom = date("Y"); echo &lt;&lt;&lt; ROBOTS User-agent: * Disallow: /cgi-bin/ Disallow: /feed/ Disallow: /wp-admin/ Disallow: /wp-content/plugins/ Disallow: /archives/ Disallow: /tag/ Disallow: /$custom Sitemap: $sitemap Crawl-delay: 4 ROBOTS; } exit; ?&gt; </code></pre>
    singulars
    1. This table or related slice is empty.
    plurals
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. This table or related slice is empty.
    1. VO
      singulars
      1. This table or related slice is empty.
    1. This table or related slice is empty.
 

Querying!

 
Guidance

SQuiL has stopped working due to an internal error.

If you are curious you may find further information in the browser console, which is accessible through the devtools (F12).

Reload