ioannis cherouvim

Suppose you have a website launched at two different hosts.

<VirtualHost *:80>
    ServerName www.example.com
    ServerAlias beta.example.com
    ....
</VirtualHost>

The content is the same but you want to serve a different robots.txt file, possibly excluding any indexing from the secondary host.

It would be handy if we could simply say:

User-agent: *
Allow: http://www.example.com/

User-agent: *
Disallow: http://beta.example.com/

to allow all bots crawl the primary host and dissalow them from the secondary one, but this syntax is imaginary. Firstly there is no Allow keyword in the spec, and secondly URLs must be relative.

The solution is to have 2 different robots.txt files:

robots-www.txt

User-agent: *
Dissalow:

robots-beta.txt

User-agent: *
Dissalow: /

and serve them via mod_rewrite like this:

<VirtualHost *:80>
    ServerName www.example.com
    ServerAlias beta.example.com
    ...
    RewriteEngine On
    RewriteCond %{HTTP_HOST} ^www\.example\.com$
    RewriteRule ^/robots.txt$ /robots-www.txt [L]
    RewriteCond %{HTTP_HOST} ^beta\.example\.com$
    RewriteRule ^/robots.txt$ /robots-beta.txt [L]
</VirtualHost>

Now http://www.example.com/robots.txt will silently serve robots-www.txt and http://beta.example.com/robots.txt will serve robots-beta.txt

This is also handy in domain name migration periods where you are waiting for dns to flush all around the globe until you feel safe for completely shutting down the secondary host and possibly assigning 301 redirects to the primary.

This entry was posted on Saturday, February 21st, 2009 at 19:06 and is filed under apache httpd, deployment. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.