Is Google going to get mad at me?
Moderator: General Moderators
- superdezign
- DevNet Master
- Posts: 4135
- Joined: Sat Jan 20, 2007 11:06 pm
Whether it's a check Google does automatically, has set intervals in which they actually take the time to check, or does out of suspicion is something I'd like to know. But regardless, Google wouldn't TELL us, they'd just scared us and have us assume that they always check. I'm sure always checking is a bit much for the spider though.
- AKA Panama Jack
- Forum Regular
- Posts: 878
- Joined: Mon Nov 14, 2005 4:21 pm
Google is by far the most persistent web crawler. They hit our game web site over a hundred times a DAY. I think half our bandwidth usage is from Google bots. I have seen up to 10 different Google Bots hit the site at the same time. Yes, 10 different IP addresses that trace back to Google.
It can get annoying at times.
A distant second is the MSNBot. It doesn't hit as often but it pulls the same image files over and over where Google apparently checks modification dates and doesn't download unchanged images.
It can get annoying at times.
A distant second is the MSNBot. It doesn't hit as often but it pulls the same image files over and over where Google apparently checks modification dates and doesn't download unchanged images.
- superdezign
- DevNet Master
- Posts: 4135
- Joined: Sat Jan 20, 2007 11:06 pm
Google accepts requests to limit the frequency of crawling. All you have to do is verify that the site is, indeed, yours and they'll do what you tell them to.AKA Panama Jack wrote:Google is by far the most persistent web crawler. They hit our game web site over a hundred times a DAY. I think half our bandwidth usage is from Google bots. I have seen up to 10 different Google Bots hit the site at the same time. Yes, 10 different IP addresses that trace back to Google.
If you go to Google Sitemaps you can adjust the voraciousness of the GoogleBot.AKA Panama Jack wrote:Google is by far the most persistent web crawler. They hit our game web site over a hundred times a DAY. I think half our bandwidth usage is from Google bots. I have seen up to 10 different Google Bots hit the site at the same time. Yes, 10 different IP addresses that trace back to Google.
It can get annoying at times.
A distant second is the MSNBot. It doesn't hit as often but it pulls the same image files over and over where Google apparently checks modification dates and doesn't download unchanged images.
- AKA Panama Jack
- Forum Regular
- Posts: 878
- Joined: Mon Nov 14, 2005 4:21 pm
I tried that many months ago. Signed up, built everything and didn't make a difference.patrikG wrote:If you go to Google Sitemaps you can adjust the voraciousness of the GoogleBot.AKA Panama Jack wrote:Google is by far the most persistent web crawler. They hit our game web site over a hundred times a DAY. I think half our bandwidth usage is from Google bots. I have seen up to 10 different Google Bots hit the site at the same time. Yes, 10 different IP addresses that trace back to Google.
It can get annoying at times.
A distant second is the MSNBot. It doesn't hit as often but it pulls the same image files over and over where Google apparently checks modification dates and doesn't download unchanged images.
- Ambush Commander
- DevNet Master
- Posts: 3698
- Joined: Mon Oct 25, 2004 9:29 pm
- Location: New Jersey, US