Get list of google search results

Ye' old general discussion board. Basically, for everything that isn't covered elsewhere. Come here to shoot the breeze, shoot your mouth off, or whatever suits your fancy.
This forum is not for asking programming related questions.

Moderator: General Moderators

Post Reply
User avatar
shiznatix
DevNet Master
Posts: 2745
Joined: Tue Dec 28, 2004 5:57 pm
Location: Tallinn, Estonia
Contact:

Get list of google search results

Post by shiznatix »

Hello kiddies. Here is my delema. My boss is hellbent on getting a list of +1000 sites that have a certain phrase in it, lets call it "beebop cola." He wants to get them from google but wants just the first link with that on it for each domain, so no domain repeating in this list. He wants to get them from google but its too much work to go through google and copy paste the links into a text file. I don't want to spider google as that is a bad idea. I have found this website: http://goohackle.com/scripts/google_parser.php which gives exactly what is needed but only returns 30 results.

We do have a custom google search engine thingy but I can't find how to search the whole internet with it, not just our website. I am not sure if its even possible to search the whole internet with it anyway.

So I ask you guys, is there a simple way to do this or not?
User avatar
onion2k
Jedi Mod
Posts: 5263
Joined: Tue Dec 21, 2004 5:03 pm
Location: usrlab.com

Re: Get list of google search results

Post by onion2k »

Use the search API - http://code.google.com/apis/ajaxsearch/documentation/ - It's geared towards their Javascript client but you can use the REST interface from PHP and parse the JSON data into something more usable pretty easily. You'll need a key - http://code.google.com/apis/ajaxsearch/signup.html
User avatar
shiznatix
DevNet Master
Posts: 2745
Joined: Tue Dec 28, 2004 5:57 pm
Location: Tallinn, Estonia
Contact:

Re: Get list of google search results

Post by shiznatix »

the problem is that it only works for sites on the domain we have so that won't do.
User avatar
Mordred
DevNet Resident
Posts: 1579
Joined: Sun Sep 03, 2006 5:19 am
Location: Sofia, Bulgaria

Re: Get list of google search results

Post by Mordred »

If you're not afraid of breaking the TOS - curl + preg_match, preferably from an IP which you won't use for a while, and with significant delays between queries.

Hint: http://www.google.com/ie?num=100&q=blah
User avatar
shiznatix
DevNet Master
Posts: 2745
Joined: Tue Dec 28, 2004 5:57 pm
Location: Tallinn, Estonia
Contact:

Re: Get list of google search results

Post by shiznatix »

aware of that, i screwed it up and got my test servers ip banned, no big deal though. i just don't want to do it to many servers of mine, because that would suck
User avatar
Mordred
DevNet Resident
Posts: 1579
Joined: Sun Sep 03, 2006 5:19 am
Location: Sofia, Bulgaria

Re: Get list of google search results

Post by Mordred »

There's such things as proxies, you know...
Post Reply