Why is our robots.txt file being ignored by the server?

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
simonmlewis
DevNet Master
Posts: 4435
Joined: Wed Oct 08, 2008 3:39 pm
Location: United Kingdom
Contact:

Why is our robots.txt file being ignored by the server?

Post by simonmlewis »

We have a site that we are developing, so we have a robots.txt file on it until we are happy it doesn't conflict with rankings of our others.

However, it doesn't read it. We see it in FTP and Plesk Filemanager, but if we go to the site ...co.uk/robots.txt, it says it cannot be found.
If I rename a file to robot.txt, that works.

Because of this, Google has cached the bloody site!!

Any ideas why a robots.txt would not work. It's a brand new file, so it not corrupted either.
Love PHP. Love CSS. Love learning new tricks too.
All the best from the United Kingdom.
User avatar
Celauran
Moderator
Posts: 6427
Joined: Tue Nov 09, 2010 2:39 pm
Location: Montreal, Canada

Re: Why is our robots.txt file being ignored by the server?

Post by Celauran »

From your .htaccess post:
[text]RewriteRule ^robots.txt$ robots.php [QSA][/text]
simonmlewis
DevNet Master
Posts: 4435
Joined: Wed Oct 08, 2008 3:39 pm
Location: United Kingdom
Contact:

Re: Why is our robots.txt file being ignored by the server?

Post by simonmlewis »

But we aren't using robots.php?
Do I need to create a PHP version to make that work - and if so, how?
Love PHP. Love CSS. Love learning new tricks too.
All the best from the United Kingdom.
User avatar
Celauran
Moderator
Posts: 6427
Joined: Tue Nov 09, 2010 2:39 pm
Location: Montreal, Canada

Re: Why is our robots.txt file being ignored by the server?

Post by Celauran »

But do you have that rule in your .htaccess? If so, there's the source of your error; it's trying to rewrite your robots.txt to a PHP file that doesn't exist.
simonmlewis
DevNet Master
Posts: 4435
Joined: Wed Oct 08, 2008 3:39 pm
Location: United Kingdom
Contact:

Re: Why is our robots.txt file being ignored by the server?

Post by simonmlewis »

Bingo. Thanks.
Love PHP. Love CSS. Love learning new tricks too.
All the best from the United Kingdom.
Post Reply