PHP Web Crawler

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
Joeiscoolone
Forum Newbie
Posts: 12
Joined: Wed May 24, 2006 5:01 pm

PHP Web Crawler

Post by Joeiscoolone »

So witch part of a PHP Web crawler is slower, the crawling the web and indexing or searching the index on the database. Witch one is faster?
User avatar
JayBird
Admin
Posts: 4524
Joined: Wed Aug 13, 2003 7:02 am
Location: York, UK
Contact:

Re: PHP Web Crawler

Post by JayBird »

crawling the web and indexing is the slowest part
Roja
Tutorials Group
Posts: 2692
Joined: Sun Jan 04, 2004 10:30 pm

Re: PHP Web Crawler

Post by Roja »

Joeiscoolone wrote:So witch part of a PHP Web crawler is slower, the crawling the web and indexing or searching the index on the database. Witch one is faster?
Generally crawling. To crawl, you have to open an http request, and then slurp the data, process it, and for every (worthwhile/working) link, you have to repeat that same process. *EACH* http request has the delays of going across the network, waiting for responses, and so on.

Searching the index is just a select. As long as the database is setup well, with lots of memory, cache, and a fast interface, possibly with clustering, it can be fairly quick.
Joeiscoolone
Forum Newbie
Posts: 12
Joined: Wed May 24, 2006 5:01 pm

Thanks

Post by Joeiscoolone »

Thanks for answering my questions.
Post Reply