Referers when google, yahoo or msn crawl ?
Moderator: General Moderators
Referers when google, yahoo or msn crawl ?
Does anyone know if google, yahoo or msn use referers or empty referers apart from user-agents when crawling our sites with a txt or xml sitemap ?
- feyd
- Neighborhood Spidermoddy
- Posts: 31559
- Joined: Mon Mar 29, 2004 3:24 pm
- Location: Bothell, Washington, USA
[url=http://forums.devnetwork.net/viewtopic.php?t=30037]Forum Rules[/url] Section 1.1 wrote:3. Do not make multiple, identical posts. This is viewed as spam and will be deleted.