PHP web crawler

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
Joeiscoolone
Forum Newbie
Posts: 12
Joined: Wed May 24, 2006 5:01 pm

PHP web crawler

Post by Joeiscoolone »

I know that you can build a PHP search engine that will crawl the web and can rank pages by keywords but how far can a PHP search engine go can it look at how many pages link to a page and rank them like that? And how many pages could a PHP search engine be designed to index?
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

  1. yes
  2. near limitless, memory withstanding
Joeiscoolone
Forum Newbie
Posts: 12
Joined: Wed May 24, 2006 5:01 pm

Thanks

Post by Joeiscoolone »

Thank you for the reply I appreciacte it.
User avatar
onion2k
Jedi Mod
Posts: 5263
Joined: Tue Dec 21, 2004 5:03 pm
Location: usrlab.com

Post by onion2k »

It's possible with PHP, but it's a very silly idea. It wouldn't scale well at all.. There are much more sensible languages for that sort of thing .. Perl for a start.
User avatar
BDKR
DevNet Resident
Posts: 1207
Joined: Sat Jun 08, 2002 1:24 pm
Location: Florida
Contact:

Post by BDKR »

onion2k wrote:It's possible with PHP, but it's a very silly idea. It wouldn't scale well at all.. There are much more sensible languages for that sort of thing .. Perl for a start.
How is Perl more scalable then PHP? Sure, there may be a performance difference in favor of Perl, but is that difference alone enough to warrant the comment made? Or wouldn't the true scalability of this utility be based on the solution and it's algorithms?
User avatar
onion2k
Jedi Mod
Posts: 5263
Joined: Tue Dec 21, 2004 5:03 pm
Location: usrlab.com

Post by onion2k »

BDKR wrote:How is Perl more scalable then PHP? Sure, there may be a performance difference in favor of Perl, but is that difference alone enough to warrant the comment made? Or wouldn't the true scalability of this utility be based on the solution and it's algorithms?
When it comes to processing text (which is basically all a search engine spider does) Perl is waaaay better than PHP (in my experience).
User avatar
BDKR
DevNet Resident
Posts: 1207
Joined: Sat Jun 08, 2002 1:24 pm
Location: Florida
Contact:

Post by BDKR »

onion2k wrote:
BDKR wrote:How is Perl more scalable then PHP? Sure, there may be a performance difference in favor of Perl, but is that difference alone enough to warrant the comment made? Or wouldn't the true scalability of this utility be based on the solution and it's algorithms?
When it comes to processing text (which is basically all a search engine spider does) Perl is waaaay better than PHP (in my experience).
That's certainly true there. Larry paid a lot of attention to that area in the earliest iterations of Perl.
Post Reply