Referers when google, yahoo or msn crawl ?

Ye' old general discussion board. Basically, for everything that isn't covered elsewhere. Come here to shoot the breeze, shoot your mouth off, or whatever suits your fancy.
This forum is not for asking programming related questions.

Moderator: General Moderators

Locked
User avatar
brolly
Forum Newbie
Posts: 18
Joined: Thu May 25, 2006 7:24 am
Location: Nevada, U.S.A.

Referers when google, yahoo or msn crawl ?

Post by brolly »

Does anyone know if google, yahoo or msn use referers or empty referers apart from user-agents when crawling our sites with a txt or xml sitemap ?
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

[url=http://forums.devnetwork.net/viewtopic.php?t=30037]Forum Rules[/url] Section 1.1 wrote:3. Do not make multiple, identical posts. This is viewed as spam and will be deleted.
Locked