Block Bots through IP adress & number of request in time
Posted: Thu Jul 03, 2008 3:46 pm
Here at my company we have this search engine for our site running on a linux server and code in PHP
I've been asked to blok spider or bots using the following aproach:
Bots and spiders are web applications that scan sites automatically, therefore request made to the site occur by hundres even thousand on a short period of time and by the same agent (same IP adress).
... So, knowing this you could say it is posible to block a bot or a spider through a PHP code that identifies when an agent or an IP adress is making too many request to your server in a really short time (which you could assume it's imposible for a human browsing your page).
I know it is posible to use PHP to collect IP adresses, write them to a text file and get them later to block that IP adress if it returns. What I don't quite know how to do is how to identify an agent through a PHP script when it has been making several request to the server in a short time. Maybe someone here could help me. Thanks
I've been asked to blok spider or bots using the following aproach:
Bots and spiders are web applications that scan sites automatically, therefore request made to the site occur by hundres even thousand on a short period of time and by the same agent (same IP adress).
... So, knowing this you could say it is posible to block a bot or a spider through a PHP code that identifies when an agent or an IP adress is making too many request to your server in a really short time (which you could assume it's imposible for a human browsing your page).
I know it is posible to use PHP to collect IP adresses, write them to a text file and get them later to block that IP adress if it returns. What I don't quite know how to do is how to identify an agent through a PHP script when it has been making several request to the server in a short time. Maybe someone here could help me. Thanks