Page 1 of 1

Crawlers and robots.txt

Posted: Thu Apr 10, 2008 8:33 am
by lafever
I was just questioning on robots.txt vs meta tags for stopping search engines from crawling certain areas of a site. From what I've been reading this is what I understand of it all.

-Using robots.txt is accessible to anyone who wants to type robots.txt in a URL and can see the directories you are trying to block
-Not all bots listen to the robots.txt of course like malware, spam, etc.
-META tags do the same thing as robots.txt but they need to be included in every page you wish not to be crawled

Basically, what I am trying to figure out is lets say there is an admin folder that I don't want to be crawled by search engine bots. Would it be a prefered method to use META tags on every page of it or include it in a robots.txt. I'm guessing not placing it in robots.txt since it is viewable to everyone.