Here at my company we have this search engine for our site running on a linux server and code in PHP
I've been asked to blok spider or bots using the following aproach:
Bots and spiders are web applications that scan sites automatically, therefore request made to the site occur by hundres even thousand on a short period of time and by the same agent (same IP adress).
... So, knowing this you could say it is posible to block a bot or a spider through a PHP code that identifies when an agent or an IP adress is making too many request to your server in a really short time (which you could assume it's imposible for a human browsing your page).
I know it is posible to use PHP to collect IP adresses, write them to a text file and get them later to block that IP adress if it returns. What I don't quite know how to do is how to identify an agent through a PHP script when it has been making several request to the server in a short time. Maybe someone here could help me. Thanks
Block Bots through IP adress & number of request in time
Moderator: General Moderators
- tecktalkcm0391
- DevNet Resident
- Posts: 1030
- Joined: Fri May 26, 2006 9:25 am
- Location: Florida
Re: Block Bots through IP adress & number of request in time
Set up a database, and record IP's (and time of request) in it... each page needs to have a code that checks the IP against the database... and if X request come back in the last X seconds/minutes... send them a 404 error or something using header()
Re: Block Bots through IP adress & number of request in time
Take a look at:
http://www.zdziarski.com/projects/mod_evasive/
http://www.zdziarski.com/projects/mod_evasive/
There are 10 types of people in this world, those who understand binary and those who don't