Page 1 of 1
PHP web crawler
Posted: Wed May 24, 2006 5:09 pm
by Joeiscoolone
I know that you can build a PHP search engine that will crawl the web and can rank pages by keywords but how far can a PHP search engine go can it look at how many pages link to a page and rank them like that? And how many pages could a PHP search engine be designed to index?
Posted: Wed May 24, 2006 5:19 pm
by feyd
- yes
- near limitless, memory withstanding
Thanks
Posted: Wed May 24, 2006 5:31 pm
by Joeiscoolone
Thank you for the reply I appreciacte it.
Posted: Thu May 25, 2006 3:03 am
by onion2k
It's possible with PHP, but it's a very silly idea. It wouldn't scale well at all.. There are much more sensible languages for that sort of thing .. Perl for a start.
Posted: Thu May 25, 2006 10:55 am
by BDKR
onion2k wrote:It's possible with PHP, but it's a very silly idea. It wouldn't scale well at all.. There are much more sensible languages for that sort of thing .. Perl for a start.
How is Perl more scalable then PHP? Sure, there may be a performance difference in favor of Perl, but is that difference alone enough to warrant the comment made? Or wouldn't the true scalability of this utility be based on the solution and it's algorithms?
Posted: Thu May 25, 2006 11:15 am
by onion2k
BDKR wrote:How is Perl more scalable then PHP? Sure, there may be a performance difference in favor of Perl, but is that difference alone enough to warrant the comment made? Or wouldn't the true scalability of this utility be based on the solution and it's algorithms?
When it comes to processing text (which is basically all a search engine spider does) Perl is waaaay better than PHP (in my experience).
Posted: Fri May 26, 2006 6:21 am
by BDKR
onion2k wrote:BDKR wrote:How is Perl more scalable then PHP? Sure, there may be a performance difference in favor of Perl, but is that difference alone enough to warrant the comment made? Or wouldn't the true scalability of this utility be based on the solution and it's algorithms?
When it comes to processing text (which is basically all a search engine spider does) Perl is waaaay better than PHP (in my experience).
That's certainly true there. Larry paid a lot of attention to that area in the earliest iterations of Perl.