PHP Web Crawler
Moderator: General Moderators
-
Joeiscoolone
- Forum Newbie
- Posts: 12
- Joined: Wed May 24, 2006 5:01 pm
PHP Web Crawler
Why is a PHP Web Crawler slower than a Web Crawler made in C++?
- tecktalkcm0391
- DevNet Resident
- Posts: 1030
- Joined: Fri May 26, 2006 9:25 am
- Location: Florida
-
Joeiscoolone
- Forum Newbie
- Posts: 12
- Joined: Wed May 24, 2006 5:01 pm
Thanks
Thank you again, I am very interested in a PHP web crawler.
- MrPotatoes
- Forum Regular
- Posts: 617
- Joined: Wed May 24, 2006 6:42 am
Re: PHP Web Crawler
The speed of any application is faaaaaar more down to the implementation than the language. A competent coder could write a fast crawler in PHP, and a poor coder would write a slow one in C++.Joeiscoolone wrote:Why is a PHP Web Crawler slower than a Web Crawler made in C++?
Merely writing your software in a different language won't instantly make it faster.
That said, some languages lend themselves to techniques that intrinsically make things quicker. C++ isn't one though .. I know for a fact that if I wrote a crawler in C++ it would be slower than one I'd write in PHP. Coz I suck at C++.
Re: PHP Web Crawler
Several reasons:Joeiscoolone wrote:Why is a PHP Web Crawler slower than a Web Crawler made in C++?
1. Overhead - Because PHP is interpreted, and run via a webserver service (generally), it has multiple layers of overhead
2. Memory limits - Web crawling requires substantial memory to parse, which is something PHP is generally limited by (8mb by default)
3. DB limits - While C can utilize the full memory of a system, PHP has only 8mb. As a result, most quasi-permanent data is stored to a database, and accessed from there - that adds I/O time and substantial overhead
4. Purpose - C is pretty much ideal for the task of large array searching, while PHP offers far less support for it.
Not to mention, the coder will make a big difference in capabilities, regardless of language.
-
Rijo Mon R
- Forum Newbie
- Posts: 1
- Joined: Thu Oct 01, 2015 1:46 am
Re: PHP Web Crawler
how to build a price comparison application.