PHP Web Crawler

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
Joeiscoolone
Forum Newbie
Posts: 12
Joined: Wed May 24, 2006 5:01 pm

PHP Web Crawler

Post by Joeiscoolone »

Why is a PHP Web Crawler slower than a Web Crawler made in C++?
User avatar
tecktalkcm0391
DevNet Resident
Posts: 1030
Joined: Fri May 26, 2006 9:25 am
Location: Florida

Post by tecktalkcm0391 »

Its because C++ is an less complex when it comes to coverting it to Binary.
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

  1. PHP is an interpreted language.
  2. It has more finite limits of memory consumption along with larger amounts of memory consumption for each variable.
  3. It's not built for performance, but more interoperability.
Joeiscoolone
Forum Newbie
Posts: 12
Joined: Wed May 24, 2006 5:01 pm

Thanks

Post by Joeiscoolone »

Thank you again, I am very interested in a PHP web crawler.
User avatar
MrPotatoes
Forum Regular
Posts: 617
Joined: Wed May 24, 2006 6:42 am

Post by MrPotatoes »

oh sweet i didn't even know that C++ was even an option for this. are there any tutorials for this? i need tomake some tools for my site to crall the website application. i thought i'd have to do it either in PHP or Perl or some other language i don't know.
User avatar
onion2k
Jedi Mod
Posts: 5263
Joined: Tue Dec 21, 2004 5:03 pm
Location: usrlab.com

Re: PHP Web Crawler

Post by onion2k »

Joeiscoolone wrote:Why is a PHP Web Crawler slower than a Web Crawler made in C++?
The speed of any application is faaaaaar more down to the implementation than the language. A competent coder could write a fast crawler in PHP, and a poor coder would write a slow one in C++.

Merely writing your software in a different language won't instantly make it faster.

That said, some languages lend themselves to techniques that intrinsically make things quicker. C++ isn't one though .. I know for a fact that if I wrote a crawler in C++ it would be slower than one I'd write in PHP. Coz I suck at C++.
Roja
Tutorials Group
Posts: 2692
Joined: Sun Jan 04, 2004 10:30 pm

Re: PHP Web Crawler

Post by Roja »

Joeiscoolone wrote:Why is a PHP Web Crawler slower than a Web Crawler made in C++?
Several reasons:

1. Overhead - Because PHP is interpreted, and run via a webserver service (generally), it has multiple layers of overhead

2. Memory limits - Web crawling requires substantial memory to parse, which is something PHP is generally limited by (8mb by default)

3. DB limits - While C can utilize the full memory of a system, PHP has only 8mb. As a result, most quasi-permanent data is stored to a database, and accessed from there - that adds I/O time and substantial overhead

4. Purpose - C is pretty much ideal for the task of large array searching, while PHP offers far less support for it.

Not to mention, the coder will make a big difference in capabilities, regardless of language.
Rijo Mon R
Forum Newbie
Posts: 1
Joined: Thu Oct 01, 2015 1:46 am

Re: PHP Web Crawler

Post by Rijo Mon R »

how to build a price comparison application.
Post Reply