Crawlers and robots.txt

Discussion of testing theory and practice, including methodologies (such as TDD, BDD, DDD, Agile, XP) and software - anything to do with testing goes here. (Formerly "The Testing Side of Development")

Moderator: General Moderators

Post Reply
User avatar
lafever
Forum Commoner
Posts: 99
Joined: Sat Apr 05, 2008 2:03 pm
Location: Taylor, MI

Crawlers and robots.txt

Post by lafever »

I was just questioning on robots.txt vs meta tags for stopping search engines from crawling certain areas of a site. From what I've been reading this is what I understand of it all.

-Using robots.txt is accessible to anyone who wants to type robots.txt in a URL and can see the directories you are trying to block
-Not all bots listen to the robots.txt of course like malware, spam, etc.
-META tags do the same thing as robots.txt but they need to be included in every page you wish not to be crawled

Basically, what I am trying to figure out is lets say there is an admin folder that I don't want to be crawled by search engine bots. Would it be a prefered method to use META tags on every page of it or include it in a robots.txt. I'm guessing not placing it in robots.txt since it is viewable to everyone.
Post Reply