PHP Web Crawler
Moderator: General Moderators
-
Joeiscoolone
- Forum Newbie
- Posts: 12
- Joined: Wed May 24, 2006 5:01 pm
PHP Web Crawler
So witch part of a PHP Web crawler is slower, the crawling the web and indexing or searching the index on the database. Witch one is faster?
Re: PHP Web Crawler
crawling the web and indexing is the slowest part
Re: PHP Web Crawler
Generally crawling. To crawl, you have to open an http request, and then slurp the data, process it, and for every (worthwhile/working) link, you have to repeat that same process. *EACH* http request has the delays of going across the network, waiting for responses, and so on.Joeiscoolone wrote:So witch part of a PHP Web crawler is slower, the crawling the web and indexing or searching the index on the database. Witch one is faster?
Searching the index is just a select. As long as the database is setup well, with lots of memory, cache, and a fast interface, possibly with clustering, it can be fairly quick.
-
Joeiscoolone
- Forum Newbie
- Posts: 12
- Joined: Wed May 24, 2006 5:01 pm
Thanks
Thanks for answering my questions.