Page 1 of 1

Bot behaviour

Posted: Fri Jan 22, 2010 8:47 am
by anon404
I am writing a script that extracts movie plots from imdb.com, the problem is i am getting denied access. Is there a way around this?
Weirdan: Source code removed

Re: Bot behaviour

Posted: Tue Jan 26, 2010 12:19 pm
by AbraCadaver
Some sites restrict access if it appears to be a bot.
Weirdan: advice removed

Re: Bot behaviour

Posted: Tue Jan 26, 2010 2:03 pm
by Weirdan
According to imdb.com terms of service you may not use any crawler unless you paid a fee to them (min 15.000USD annually, details here: http://www.imdb.com/licensing/). If you did, you would be using their webservice to get the data you need, not a crawler like this.
They allow limited non-commercial use of their data in personal projects, and that data is provided via ftp (details here: http://www.imdb.com/interfaces)

Therefore this crawler clearly qualifies as a program for producing illegal copies of copyrighted content.

Source code removed, original post is locked. You may continue to discuss legal ways to use imdb content here though.

AbraCadaver, you get a verbal warning for unknowingly engaging in an activity that is against forum rules.