Page 1 of 1

Indexing Sites...

Posted: Mon Oct 06, 2003 3:55 am
by Mr. Tech
I'm wanting to set up a script which indexes 100's even 1000's of different websites. This is the code I will use:

Code: Select all

<?php
$query = "select * from websites";
$result = mysql_query($query);
$numrows = mysql_num_rows($result);
for($x=0;$x<$numrows;$x++) {
	$row = mysql_fetch_assoc($result);

$fp = @fopen($row[url], "r" ); // Open the URL
      $content = "";
      if( $fp )
      {
        // Read in the URL
        while( !feof( $fp ) )
          $content .= fread( $fp, 10240 );        
        fclose( $fp );
       }
}
?>
Now, if it is indexing 1000's sites at one time, it will time out right? is there anyway to avoide that?

Also, if I use a cron job to run the script, can that time out?

Thanks

Posted: Mon Oct 06, 2003 8:33 am
by JAM
1. Timeouts general

Code: Select all

// php.ini
// for sockets, but I'll mention it as you could change from fopen to fsockopen()
default_socket_timeout = 60
/ more interesting...
max_execution_time = 30     ; Maximum execution time of each script, in seconds

Code: Select all

<?php 
// example to change the above
    ini_set("max_execution_time", "60000000");  
?>
2. Cronjob timing out
I belive it can. If you run this script 1 time/h, you could add a verifying-script that looks at if all 1000 sites where indexed. If not, run the ordinary script once again, but let that start from url #<whatever>.

(Hope I was clear enough...)

Posted: Mon Oct 06, 2003 9:19 am
by Cruzado_Mainfrm
you can add:

Code: Select all

<?php
set_time_limit(0);
?>
at the start of your script and it will never time out :D

Posted: Mon Oct 06, 2003 5:00 pm
by Mr. Tech
Cool! Thanks!