All-In-One Search Engines - Do they steal result pages?

Ye' old general discussion board. Basically, for everything that isn't covered elsewhere. Come here to shoot the breeze, shoot your mouth off, or whatever suits your fancy.
This forum is not for asking programming related questions.

Moderator: General Moderators

Post Reply
kaYak
Forum Commoner
Posts: 65
Joined: Mon Feb 02, 2004 2:43 pm
Location: USA

All-In-One Search Engines - Do they steal result pages?

Post by kaYak »

Looking at DogPile.com I was wondering if it just steals results from the various sources and compiles them into one list of results just by grabbing the content of search engine result pages. When processing your search does their script visit Google, Ask, Yahoo, and whatever else and just pull the results from the HTML code and parse it accordingly? It doesn't seem like this would be legal if they did. Could anyone shed some light on this?
LiveFree
Forum Contributor
Posts: 258
Joined: Tue Dec 06, 2005 5:34 pm
Location: W-Town

Post by LiveFree »

If those multi-engine searchers are, they could get into some big <span style='color:blue' title='I&#39;m naughty, are you naughty?'>smurf</span>.
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

I believe most of them do. By permission or not, I can't say. They may also cache the resultant data to ease the bandwidth needs.
kaYak
Forum Commoner
Posts: 65
Joined: Mon Feb 02, 2004 2:43 pm
Location: USA

Post by kaYak »

I thought it would be interesting to create a mini-search engine and then I see that some "search engines" just gather them from other engines, so it made me wonder. I could pretty easily make something in php to steal content from Google and parse it, although it may be easy it's most likely not legal, let alone right.
User avatar
Burrito
Spockulator
Posts: 4715
Joined: Wed Feb 04, 2004 8:15 pm
Location: Eden, Utah

Post by Burrito »

what would be the point of that anyway?
kaYak
Forum Commoner
Posts: 65
Joined: Mon Feb 02, 2004 2:43 pm
Location: USA

Post by kaYak »

The point of stealing from Google?
User avatar
Burrito
Spockulator
Posts: 4715
Joined: Wed Feb 04, 2004 8:15 pm
Location: Eden, Utah

Post by Burrito »

no, the point of creating a search engine that just displays results from google...why not just search on google?
User avatar
daedalus__
DevNet Resident
Posts: 1925
Joined: Thu Feb 09, 2006 4:52 pm

Post by daedalus__ »

Selling ads on your search and giving yourself money rather than Google.
User avatar
John Cartwright
Site Admin
Posts: 11470
Joined: Tue Dec 23, 2003 2:10 am
Location: Toronto
Contact:

Post by John Cartwright »

Burrito means, if I want to search google why would I go to some other site to google when I can just go to the real thing?
User avatar
Jenk
DevNet Master
Posts: 3587
Joined: Mon Sep 19, 2005 6:24 am
Location: London

Post by Jenk »

These sites prey on those who are unaware of googles existance, some claiming to use multiple search sites prey on those too lazy to use them individually, or just to get the "wow, they search on all of those and put into one page!!11" factor.

The money is in the advertisement. It is also a ploy to track peoples search habits, find out what is popular, then increase advertising prices on the popular pages.
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

Well, google "offer" their search services to other companies. From our point of view their are literally thousands of search engines (webistes). In reality, there are only a handful. Most search engine sites are using a more commercial source for their results.

http://searchenginewatch.com/showPage.html?page=2156401
kaYak
Forum Commoner
Posts: 65
Joined: Mon Feb 02, 2004 2:43 pm
Location: USA

Post by kaYak »

Google offers their search to webmasters who wish to place on their site either a search form which links to Google, or an offsite semi-branded page. They also offer their API with limited queries per day and with limited non-commercial use. Beyond that they only offer partnered search options to large, high traffic services such as AOL (and Google has stake in AOL). Perhaps sites like DogPile have an agreement with the big search engines that they will attract the users who like the combined search, and give money to the engines they grab results from, and make some money themselves? I'm not sure, but I'm curious. I can't imagine them just pulling results without some sort of agreement.

EDIT: The search engine watch page does make it clear that they have partnerships and it appears that they buy the results from the major search engines.
User avatar
Ambush Commander
DevNet Master
Posts: 3698
Joined: Mon Oct 25, 2004 9:29 pm
Location: New Jersey, US

Post by Ambush Commander »

Search engines are all about making information you want to find easier to find. Even ripping off results from multiple search engines might be effective... if it turns out that you're more likely to get useful results on those engines.

Nevertheless, I have never used dogpile before (although I have had it recommended to me before).
User avatar
RobertGonzalez
Site Administrator
Posts: 14293
Joined: Tue Sep 09, 2003 6:04 pm
Location: Fremont, CA, USA

Post by RobertGonzalez »

There is nothing that compares to google, in my opinion. The only thing that comes close is dmoz, but that doesn't have the sponsored listings so people that actually are willing to spend money to reach can't reach you on it.

As for who gets the money from advertising, whether someone pays google for you to find them, or someone pays you so you can pay google to help you let the user find the payer, google still inherits the earth. Their site is easy to use, so I use it and don't plan on going to a combination, multi-search engine site any time soon.
Post Reply