Page 2 of 2

Posted: Mon Jan 05, 2004 8:36 pm
by JAM
http://www.robotstxt.org/wc/robots.html
For more info about webspiders/robots and the robot.txt if anyone is interested.