Page 1 of 1

Restrict spiders to low-demand times?

Posted: Wed Dec 10, 2008 3:02 pm
by intellivision
Is there a syntax to place in robots.txt to restrict (ask, prod) spiders to visit at certain hours?

I know there's a page request frequency syntax, but certain hours would work better for my server. It's getting kinda taxed.

Re: Restrict spiders to low-demand times?

Posted: Wed Dec 10, 2008 6:23 pm
by Peter Anselmo
Yes, there is, here's an example robots.txt file:

Code: Select all

User-agent: *
Disallow: 
Request-rate: 1/5         # maximum rate is one page every 5 seconds
Visit-time: 0600-0845     # only visit between 06:00 and 08:45 UTC (GMT)
 
Wikipedia also has lots of detailed info here

Keep in mind the robots.txt is just a suggestion to compliant crawlers, spam bots won't check or care.