Comments on: robots.txt control for host aliases via mod_rewrite

By: cherouvim

cherouvim — Tue, 28 Apr 2009 05:57:19 +0000

@Israel WebDev: I think plain “Dissalow:” is OK as well.

@Neil Ferns: The virtual host in this example is 1 with 2 domains (aliases).

By: Neil Ferns

Neil Ferns — Tue, 28 Apr 2009 05:50:37 +0000

We could use a
Alias /robots.txt /home/www/www.example.com/htdocs/noallow-robots.txt
which could prevent complicated mod rewrite rules.

and have 2 diff virtaul hosts for the 2 diff domains pointing to the same htdocs.
Alias /robots.txt /home/www/www.example.com/htdocs/Allow-robots.txt
or no file.

By: Israel WebDev

Israel WebDev — Tue, 21 Apr 2009 20:39:14 +0000

This is perfect! Thanks.
Small note: the robots-beta.txt should read

User-agent: *
Disallow: /

and robots-www.txt should be:

User-agent: *

or a blank file (, or no file)