Hello kiddies. Here is my delema. My boss is hellbent on getting a list of +1000 sites that have a certain phrase in it, lets call it "beebop cola." He wants to get them from google but wants just the first link with that on it for each domain, so no domain repeating in this list. He wants to get them from google but its too much work to go through google and copy paste the links into a text file. I don't want to spider google as that is a bad idea. I have found this website: http://goohackle.com/scripts/google_parser.php which gives exactly what is needed but only returns 30 results.
We do have a custom google search engine thingy but I can't find how to search the whole internet with it, not just our website. I am not sure if its even possible to search the whole internet with it anyway.
So I ask you guys, is there a simple way to do this or not?
Get list of google search results
Moderator: General Moderators
Re: Get list of google search results
Use the search API - http://code.google.com/apis/ajaxsearch/documentation/ - It's geared towards their Javascript client but you can use the REST interface from PHP and parse the JSON data into something more usable pretty easily. You'll need a key - http://code.google.com/apis/ajaxsearch/signup.html
- shiznatix
- DevNet Master
- Posts: 2745
- Joined: Tue Dec 28, 2004 5:57 pm
- Location: Tallinn, Estonia
- Contact:
Re: Get list of google search results
the problem is that it only works for sites on the domain we have so that won't do.
Re: Get list of google search results
If you're not afraid of breaking the TOS - curl + preg_match, preferably from an IP which you won't use for a while, and with significant delays between queries.
Hint: http://www.google.com/ie?num=100&q=blah
Hint: http://www.google.com/ie?num=100&q=blah
- shiznatix
- DevNet Master
- Posts: 2745
- Joined: Tue Dec 28, 2004 5:57 pm
- Location: Tallinn, Estonia
- Contact:
Re: Get list of google search results
aware of that, i screwed it up and got my test servers ip banned, no big deal though. i just don't want to do it to many servers of mine, because that would suck
Re: Get list of google search results
There's such things as proxies, you know...