Page 1 of 1

PHP Web Crawler

Posted: Fri Jun 02, 2006 6:01 am
by Joeiscoolone
So witch part of a PHP Web crawler is slower, the crawling the web and indexing or searching the index on the database. Witch one is faster?

Re: PHP Web Crawler

Posted: Fri Jun 02, 2006 6:20 am
by JayBird
crawling the web and indexing is the slowest part

Re: PHP Web Crawler

Posted: Fri Jun 02, 2006 8:23 am
by Roja
Joeiscoolone wrote:So witch part of a PHP Web crawler is slower, the crawling the web and indexing or searching the index on the database. Witch one is faster?
Generally crawling. To crawl, you have to open an http request, and then slurp the data, process it, and for every (worthwhile/working) link, you have to repeat that same process. *EACH* http request has the delays of going across the network, waiting for responses, and so on.

Searching the index is just a select. As long as the database is setup well, with lots of memory, cache, and a fast interface, possibly with clustering, it can be fairly quick.

Thanks

Posted: Sun Jun 04, 2006 11:16 pm
by Joeiscoolone
Thanks for answering my questions.