Page 1 of 1

Get list of google search results

Posted: Wed Sep 03, 2008 2:16 am
by shiznatix
Hello kiddies. Here is my delema. My boss is hellbent on getting a list of +1000 sites that have a certain phrase in it, lets call it "beebop cola." He wants to get them from google but wants just the first link with that on it for each domain, so no domain repeating in this list. He wants to get them from google but its too much work to go through google and copy paste the links into a text file. I don't want to spider google as that is a bad idea. I have found this website: http://goohackle.com/scripts/google_parser.php which gives exactly what is needed but only returns 30 results.

We do have a custom google search engine thingy but I can't find how to search the whole internet with it, not just our website. I am not sure if its even possible to search the whole internet with it anyway.

So I ask you guys, is there a simple way to do this or not?

Re: Get list of google search results

Posted: Wed Sep 03, 2008 2:58 am
by onion2k
Use the search API - http://code.google.com/apis/ajaxsearch/documentation/ - It's geared towards their Javascript client but you can use the REST interface from PHP and parse the JSON data into something more usable pretty easily. You'll need a key - http://code.google.com/apis/ajaxsearch/signup.html

Re: Get list of google search results

Posted: Thu Sep 04, 2008 5:52 am
by shiznatix
the problem is that it only works for sites on the domain we have so that won't do.

Re: Get list of google search results

Posted: Thu Sep 04, 2008 7:25 am
by Mordred
If you're not afraid of breaking the TOS - curl + preg_match, preferably from an IP which you won't use for a while, and with significant delays between queries.

Hint: http://www.google.com/ie?num=100&q=blah

Re: Get list of google search results

Posted: Thu Sep 04, 2008 8:44 am
by shiznatix
aware of that, i screwed it up and got my test servers ip banned, no big deal though. i just don't want to do it to many servers of mine, because that would suck

Re: Get list of google search results

Posted: Thu Sep 04, 2008 9:21 am
by Mordred
There's such things as proxies, you know...