Jan 102018
 

As I planned move of my site to Linode, first I needed a place to test. It was easy enough to create test domain and fill it with migrated data but I didn’t want Google (or any other bot) to index it. The easiest way to do so was to require authentication. In Apache configuration that can be done using Directory directive:

<Directory "/var/www/html">
    AuthType Basic
    AuthUserFile "/var/www/.htpasswd"
    Require valid-user
</Directory>

However, this also means that my robots.txt with disallow statements was also forbidden. What I really wanted was to allow only access to robots.txt while forbidding everything else.

A bit of modification later, this is what I came up with:

<Directory "/var/www/html">
    AuthType Basic
    AuthUserFile "/var/www/.htpasswd"
    Require valid-user
    <Files "robots.txt">
        Allow from all
        Satisfy Any
    </Files>
</Directory>

 Leave a Reply

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>

This site uses Akismet to reduce spam. Learn how your comment data is processed.