Posted: Wed Jun 20, 2007 10:38 pm
I've read that in many places. The page I posted more or less said that (except without explicitly stating that they parse css)
A community of PHP developers offering assistance, advice, discussion, and friendship.
http://forums.devnetwork.net/
Google accepts requests to limit the frequency of crawling. All you have to do is verify that the site is, indeed, yours and they'll do what you tell them to.AKA Panama Jack wrote:Google is by far the most persistent web crawler. They hit our game web site over a hundred times a DAY. I think half our bandwidth usage is from Google bots. I have seen up to 10 different Google Bots hit the site at the same time. Yes, 10 different IP addresses that trace back to Google.
If you go to Google Sitemaps you can adjust the voraciousness of the GoogleBot.AKA Panama Jack wrote:Google is by far the most persistent web crawler. They hit our game web site over a hundred times a DAY. I think half our bandwidth usage is from Google bots. I have seen up to 10 different Google Bots hit the site at the same time. Yes, 10 different IP addresses that trace back to Google.
It can get annoying at times.
A distant second is the MSNBot. It doesn't hit as often but it pulls the same image files over and over where Google apparently checks modification dates and doesn't download unchanged images.
I tried that many months ago. Signed up, built everything and didn't make a difference.patrikG wrote:If you go to Google Sitemaps you can adjust the voraciousness of the GoogleBot.AKA Panama Jack wrote:Google is by far the most persistent web crawler. They hit our game web site over a hundred times a DAY. I think half our bandwidth usage is from Google bots. I have seen up to 10 different Google Bots hit the site at the same time. Yes, 10 different IP addresses that trace back to Google.
It can get annoying at times.
A distant second is the MSNBot. It doesn't hit as often but it pulls the same image files over and over where Google apparently checks modification dates and doesn't download unchanged images.