Long running script freaks out

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
mazzitech
Forum Newbie
Posts: 4
Joined: Tue Dec 13, 2011 8:38 pm

Long running script freaks out

Post by mazzitech »

I am attempting to read a large file, about 350Mb with about 650,000 records. The load program started acting up so I am in debug mode.

I wrote a short script to simply count the records with fgetcsv, which is what the load uses. I run the script thru a browser. When I read anywhere 15,000 to 30,000 records, the counters that are being displayed go crazy, they go from 30, 000 to 700, then continue counting then they jump to 32,000, count some more then drop back to a lower number then count some more for a very long time. I clocked the counter at over 3 minutes (I got past the time out issues) there are no PHP warnings.

Now the interesting part. If I insert an echo "x"; statement into the read loop, the script prints out the list of x's on the screen, the counters work as expected and time drops down to about 30 seconds. Same file and no other changes to the script.

It looks like the script is being swapped in and out with the variables being reset with random data. I'm not doing anything with memory management. It also does something else strange. If, while the script is running, I make a change to the script, the new script is sometimes executed (I added an echo statement that would run at the end, and it was executed)

Does anyone have a clue why this is happening?
Last edited by mazzitech on Wed Dec 14, 2011 7:41 pm, edited 1 time in total.
mikeashfield
Forum Contributor
Posts: 159
Joined: Sat Oct 22, 2011 10:50 am

Re: Long running script freaks out

Post by mikeashfield »

replace:

Code: Select all

echo x;
with:

Code: Select all

x=x;
And see if that has any effect? :)
mikeashfield
Forum Contributor
Posts: 159
Joined: Sat Oct 22, 2011 10:50 am

Re: Long running script freaks out

Post by mikeashfield »

Failing that, do something similar to this:

Code: Select all

<?php
x = 1;
for (i = 1; i < 100; i++) {
y = x;
echo x;
x = y++;
}
?>
User avatar
McInfo
DevNet Resident
Posts: 1532
Joined: Wed Apr 01, 2009 1:31 pm

Re: Long running script freaks out

Post by McInfo »

mazzitech wrote:Does anyone have a clue why this is happening?
It might be easier for us to diagnose the problem if we could see the script.
mazzitech
Forum Newbie
Posts: 4
Joined: Tue Dec 13, 2011 8:38 pm

Re: Long running script freaks out

Post by mazzitech »

The counting program did a few more things, such as counting (cnt++;) and it was writing a count record to a file (fopen, fwrite, fclose) on every iteration. I had another script reading that fie every 10 secs that displayed the counts. I had no need for the echo "x"; Changing the character from an X to a blank had no effect, the script still ran correctly. I tried putting that into the load script where the problem first surfaced and it still doesn't work right.
mazzitech
Forum Newbie
Posts: 4
Joined: Tue Dec 13, 2011 8:38 pm

Re: Long running script freaks out

Post by mazzitech »

Here is the script

Code: Select all

<?php 
   error_reporting(E_ALL);
   ini_set("max_execution_time", "3000");
   ini_set('memory_limit','640M');
   set_time_limit(2500);                                                                    // set the max time for the script, 30 secs = 80,000 records
   $cnt=0;
   $cntin=0;
   $loop=0;
   $sw=0;
   $c=0;
   $maxlen=0;
     $fhcnt = fopen("readcount.log", 'w');
	 fclose($fhcnt);
   $fhin = fopen("selected_npidata_20111114.csv", 'r') or die("Failed to open Input file");    // NPI INput file
//   echo $fhin;
   $header = fgetcsv($fhin, 10000, ",");                                                  // Read the header for CB

   while (($data = fgets($fhin, 10000)) !== FALSE)                                // read and convert the NPI data for CB                         
    {
      $cntin++;
      $c++;
	  $hold = strlen($data);
	  if ($hold > $maxlen) {
		  echo "<p>".$hold."</p>";
		  $maxlen = $hold;
	  }
//      echo " ";
	  $fhcnt = fopen("readcount.log", 'r+');
	  fwrite($fhcnt,"Record number ".$cntin." Max Length: ".$maxlen."<br />"); 
	  fclose($fhcnt);  
	  if (file_exists("stop.log")) {
		  break;			
	  }
   }
 
   fclose($fhin);
   $fhcnt = fopen("readcount.log", 'r+');
   fwrite($fhcnt,"Finished - Counted ".$cntin." records, Max Length".$maxlen); 
   fclose($fhcnt);
   echo "<p>Finished!, Counted $cntin records Max Length: $maxlen</p>";
Last edited by Benjamin on Fri Dec 23, 2011 2:18 am, edited 1 time in total.
Reason: Added [syntax=php||htm||css||javascript||sql||etc] - Please use [syntax] tags when posting code in the forums! Thanks.
mazzitech
Forum Newbie
Posts: 4
Joined: Tue Dec 13, 2011 8:38 pm

Re: Long running script freaks out

Post by mazzitech »

Solved! The culprit here is the load balancing scheme that the host is using. When the echo was used, the "sticky sessions" capability came into play. Without that, the load balancer didn't work properly. In order to get enough execution time, the script needs to be run by cron
User avatar
tr0gd0rr
Forum Contributor
Posts: 305
Joined: Thu May 11, 2006 8:58 pm
Location: Utah, USA

Re: Long running script freaks out

Post by tr0gd0rr »

Interesting. Thanks for sharing the solution.
Post Reply