Posted: Mon Jan 05, 2004 8:36 pm
http://www.robotstxt.org/wc/robots.html
For more info about webspiders/robots and the robot.txt if anyone is interested.
For more info about webspiders/robots and the robot.txt if anyone is interested.