Page 1 of 1

Why is our robots.txt file being ignored by the server?

Posted: Tue Apr 07, 2015 9:24 am
by simonmlewis
We have a site that we are developing, so we have a robots.txt file on it until we are happy it doesn't conflict with rankings of our others.

However, it doesn't read it. We see it in FTP and Plesk Filemanager, but if we go to the site ...co.uk/robots.txt, it says it cannot be found.
If I rename a file to robot.txt, that works.

Because of this, Google has cached the bloody site!!

Any ideas why a robots.txt would not work. It's a brand new file, so it not corrupted either.

Re: Why is our robots.txt file being ignored by the server?

Posted: Tue Apr 07, 2015 9:26 am
by Celauran
From your .htaccess post:
[text]RewriteRule ^robots.txt$ robots.php [QSA][/text]

Re: Why is our robots.txt file being ignored by the server?

Posted: Tue Apr 07, 2015 9:32 am
by simonmlewis
But we aren't using robots.php?
Do I need to create a PHP version to make that work - and if so, how?

Re: Why is our robots.txt file being ignored by the server?

Posted: Tue Apr 07, 2015 9:34 am
by Celauran
But do you have that rule in your .htaccess? If so, there's the source of your error; it's trying to rewrite your robots.txt to a PHP file that doesn't exist.

Re: Why is our robots.txt file being ignored by the server?

Posted: Tue Apr 07, 2015 9:38 am
by simonmlewis
Bingo. Thanks.