Any one know how to search faster?

Not for 'how-to' coding questions but PHP theory instead, this forum is here for those of us who wish to learn about design aspects of programming with PHP.

Moderator: General Moderators

Post Reply
Nashtrump
Forum Newbie
Posts: 13
Joined: Wed May 03, 2006 1:56 pm

Any one know how to search faster?

Post by Nashtrump »

hi everyone...

Im fairly new to PHP and am developing a program that looks for certain things in other websites.

Currently imn using the file_get_contents($link) to look at each url.

This is really slow though... does anyone have any better suggestions to do this?
Im aiming to look at about 5-6 websites each containing around 30-40 urls...

Any help here would be really appreicated!!

Thank you in advance!!

Nash
User avatar
Christopher
Site Administrator
Posts: 13596
Joined: Wed Aug 25, 2004 7:54 pm
Location: New York, NY, US

Post by Christopher »

Have you Googled "php web spider" to see how others have done it. There are also several very good packages already written to do this.
(#10850)
Nashtrump
Forum Newbie
Posts: 13
Joined: Wed May 03, 2006 1:56 pm

Post by Nashtrump »

Hi there,

I am looking for a specific thing to spider so unfortunately for me its a spider that must be manually built. So i cant use any other packages.

I can only find one or two examples of spiders and they both use the file_get_contents.

I just wondered if anyone knew of any other way which was quicker?

thanks

nash
User avatar
Ollie Saunders
DevNet Master
Posts: 3179
Joined: Tue May 24, 2005 6:01 pm
Location: UK

Post by Ollie Saunders »

well its always going to be limited by the speed of your internet connection and the speed at which the websites can respond.

Have a look at declare(ticks) which may help to address that second bottleneck i mentions by requesting data from multiple sites at once. Although the comments about declare ticks don't sound very inspiring
sava
Forum Newbie
Posts: 8
Joined: Mon May 15, 2006 10:04 am

Post by sava »

file_get_contents($link) retrives all $link content. May be it's better to use the stream and read the $line while you find what you are looking for...

Code: Select all

$fp = fopen($link,'r');
$str = '';
while (!feof($fp) && !substr_count($str,$youSearch)) {
	$str = fread($fp,1024);
}
Post Reply