Page 1 of 1

Referers when google, yahoo or msn crawl ?

Posted: Sun Jun 18, 2006 9:40 am
by brolly
Does anyone know if google, yahoo or msn use referers or empty referers apart from user-agents when crawling our sites with a txt or xml sitemap ?

Posted: Sun Jun 18, 2006 3:32 pm
by feyd
[url=http://forums.devnetwork.net/viewtopic.php?t=30037]Forum Rules[/url] Section 1.1 wrote:3. Do not make multiple, identical posts. This is viewed as spam and will be deleted.