Restrict spiders to low-demand times?
Posted: Wed Dec 10, 2008 3:02 pm
Is there a syntax to place in robots.txt to restrict (ask, prod) spiders to visit at certain hours?
I know there's a page request frequency syntax, but certain hours would work better for my server. It's getting kinda taxed.
I know there's a page request frequency syntax, but certain hours would work better for my server. It's getting kinda taxed.