detect searchengine-bot

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
User avatar
potato
Forum Contributor
Posts: 192
Joined: Tue Mar 16, 2004 8:30 am
Location: my lovely trailer, next to the big tree

detect searchengine-bot

Post by potato »

Hi,

i'm looking for the best way to detect a searchengine-bot, for example if google or yahoo passes to crawl my site.

Anyone has some experience with this?

greetings, tom
User avatar
volka
DevNet Evangelist
Posts: 8391
Joined: Tue May 07, 2002 9:48 am
Location: Berlin, ger

Post by volka »

You can examine the user-agent string (e.g. Googlebot) and (maybe) the ip range.
User avatar
aaronhall
DevNet Resident
Posts: 1040
Joined: Tue Aug 13, 2002 5:10 pm
Location: Back in Phoenix, missing the microbrews
Contact:

Post by aaronhall »

The user agent is available with $_SERVER['HTTP_USER_AGENT']. A list of web spider user agents is available at http://www.psychedelix.com/agents/index.shtml or http://www.psychedelix.com/agents/index.shtml
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

get_browser() can often be a simple and fast way to determine if the requesting agent suggests further digging.
Post Reply