Indexing Sites...

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
Mr. Tech
Forum Contributor
Posts: 205
Joined: Tue Feb 11, 2003 4:18 pm
Location: Australia

Indexing Sites...

Post by Mr. Tech »

I'm wanting to set up a script which indexes 100's even 1000's of different websites. This is the code I will use:

Code: Select all

<?php
$query = "select * from websites";
$result = mysql_query($query);
$numrows = mysql_num_rows($result);
for($x=0;$x<$numrows;$x++) {
	$row = mysql_fetch_assoc($result);

$fp = @fopen($row[url], "r" ); // Open the URL
      $content = "";
      if( $fp )
      {
        // Read in the URL
        while( !feof( $fp ) )
          $content .= fread( $fp, 10240 );        
        fclose( $fp );
       }
}
?>
Now, if it is indexing 1000's sites at one time, it will time out right? is there anyway to avoide that?

Also, if I use a cron job to run the script, can that time out?

Thanks
User avatar
JAM
DevNet Resident
Posts: 2101
Joined: Fri Aug 08, 2003 6:53 pm
Location: Sweden
Contact:

Post by JAM »

1. Timeouts general

Code: Select all

// php.ini
// for sockets, but I'll mention it as you could change from fopen to fsockopen()
default_socket_timeout = 60
/ more interesting...
max_execution_time = 30     ; Maximum execution time of each script, in seconds

Code: Select all

<?php 
// example to change the above
    ini_set("max_execution_time", "60000000");  
?>
2. Cronjob timing out
I belive it can. If you run this script 1 time/h, you could add a verifying-script that looks at if all 1000 sites where indexed. If not, run the ordinary script once again, but let that start from url #<whatever>.

(Hope I was clear enough...)
Cruzado_Mainfrm
Forum Contributor
Posts: 346
Joined: Sun Jun 15, 2003 11:22 pm
Location: Miami, FL

Post by Cruzado_Mainfrm »

you can add:

Code: Select all

<?php
set_time_limit(0);
?>
at the start of your script and it will never time out :D
Mr. Tech
Forum Contributor
Posts: 205
Joined: Tue Feb 11, 2003 4:18 pm
Location: Australia

Post by Mr. Tech »

Cool! Thanks!
Post Reply