I was just questioning on robots.txt vs meta tags for stopping search engines from crawling certain areas of a site. From what I've been reading this is what I understand of it all.
-Using robots.txt is accessible to anyone who wants to type robots.txt in a URL and can see the directories you are trying to block
-Not all bots listen to the robots.txt of course like malware, spam, etc.
-META tags do the same thing as robots.txt but they need to be included in every page you wish not to be crawled
Basically, what I am trying to figure out is lets say there is an admin folder that I don't want to be crawled by search engine bots. Would it be a prefered method to use META tags on every page of it or include it in a robots.txt. I'm guessing not placing it in robots.txt since it is viewable to everyone.
Crawlers and robots.txt
Moderator: General Moderators