PHP Web Crawler
Posted: Fri Jun 02, 2006 6:01 am
So witch part of a PHP Web crawler is slower, the crawling the web and indexing or searching the index on the database. Witch one is faster?
A community of PHP developers offering assistance, advice, discussion, and friendship.
http://forums.devnetwork.net/
Generally crawling. To crawl, you have to open an http request, and then slurp the data, process it, and for every (worthwhile/working) link, you have to repeat that same process. *EACH* http request has the delays of going across the network, waiting for responses, and so on.Joeiscoolone wrote:So witch part of a PHP Web crawler is slower, the crawling the web and indexing or searching the index on the database. Witch one is faster?