Page 1 of 1

Long running script freaks out

Posted: Tue Dec 13, 2011 8:56 pm
by mazzitech
I am attempting to read a large file, about 350Mb with about 650,000 records. The load program started acting up so I am in debug mode.

I wrote a short script to simply count the records with fgetcsv, which is what the load uses. I run the script thru a browser. When I read anywhere 15,000 to 30,000 records, the counters that are being displayed go crazy, they go from 30, 000 to 700, then continue counting then they jump to 32,000, count some more then drop back to a lower number then count some more for a very long time. I clocked the counter at over 3 minutes (I got past the time out issues) there are no PHP warnings.

Now the interesting part. If I insert an echo "x"; statement into the read loop, the script prints out the list of x's on the screen, the counters work as expected and time drops down to about 30 seconds. Same file and no other changes to the script.

It looks like the script is being swapped in and out with the variables being reset with random data. I'm not doing anything with memory management. It also does something else strange. If, while the script is running, I make a change to the script, the new script is sometimes executed (I added an echo statement that would run at the end, and it was executed)

Does anyone have a clue why this is happening?

Re: Long running script freaks out

Posted: Wed Dec 14, 2011 5:03 am
by mikeashfield
replace:

Code: Select all

echo x;
with:

Code: Select all

x=x;
And see if that has any effect? :)

Re: Long running script freaks out

Posted: Wed Dec 14, 2011 10:49 am
by mikeashfield
Failing that, do something similar to this:

Code: Select all

<?php
x = 1;
for (i = 1; i < 100; i++) {
y = x;
echo x;
x = y++;
}
?>

Re: Long running script freaks out

Posted: Wed Dec 14, 2011 11:19 am
by McInfo
mazzitech wrote:Does anyone have a clue why this is happening?
It might be easier for us to diagnose the problem if we could see the script.

Re: Long running script freaks out

Posted: Wed Dec 14, 2011 12:03 pm
by mazzitech
The counting program did a few more things, such as counting (cnt++;) and it was writing a count record to a file (fopen, fwrite, fclose) on every iteration. I had another script reading that fie every 10 secs that displayed the counts. I had no need for the echo "x"; Changing the character from an X to a blank had no effect, the script still ran correctly. I tried putting that into the load script where the problem first surfaced and it still doesn't work right.

Re: Long running script freaks out

Posted: Sat Dec 17, 2011 2:15 pm
by mazzitech
Here is the script

Code: Select all

<?php 
   error_reporting(E_ALL);
   ini_set("max_execution_time", "3000");
   ini_set('memory_limit','640M');
   set_time_limit(2500);                                                                    // set the max time for the script, 30 secs = 80,000 records
   $cnt=0;
   $cntin=0;
   $loop=0;
   $sw=0;
   $c=0;
   $maxlen=0;
     $fhcnt = fopen("readcount.log", 'w');
	 fclose($fhcnt);
   $fhin = fopen("selected_npidata_20111114.csv", 'r') or die("Failed to open Input file");    // NPI INput file
//   echo $fhin;
   $header = fgetcsv($fhin, 10000, ",");                                                  // Read the header for CB

   while (($data = fgets($fhin, 10000)) !== FALSE)                                // read and convert the NPI data for CB                         
    {
      $cntin++;
      $c++;
	  $hold = strlen($data);
	  if ($hold > $maxlen) {
		  echo "<p>".$hold."</p>";
		  $maxlen = $hold;
	  }
//      echo " ";
	  $fhcnt = fopen("readcount.log", 'r+');
	  fwrite($fhcnt,"Record number ".$cntin." Max Length: ".$maxlen."<br />"); 
	  fclose($fhcnt);  
	  if (file_exists("stop.log")) {
		  break;			
	  }
   }
 
   fclose($fhin);
   $fhcnt = fopen("readcount.log", 'r+');
   fwrite($fhcnt,"Finished - Counted ".$cntin." records, Max Length".$maxlen); 
   fclose($fhcnt);
   echo "<p>Finished!, Counted $cntin records Max Length: $maxlen</p>";

Re: Long running script freaks out

Posted: Wed Dec 21, 2011 10:32 pm
by mazzitech
Solved! The culprit here is the load balancing scheme that the host is using. When the echo was used, the "sticky sessions" capability came into play. Without that, the load balancer didn't work properly. In order to get enough execution time, the script needs to be run by cron

Re: Long running script freaks out

Posted: Thu Dec 22, 2011 1:56 pm
by tr0gd0rr
Interesting. Thanks for sharing the solution.