Long running script freaks out
Moderator: General Moderators
Long running script freaks out
I am attempting to read a large file, about 350Mb with about 650,000 records. The load program started acting up so I am in debug mode.
I wrote a short script to simply count the records with fgetcsv, which is what the load uses. I run the script thru a browser. When I read anywhere 15,000 to 30,000 records, the counters that are being displayed go crazy, they go from 30, 000 to 700, then continue counting then they jump to 32,000, count some more then drop back to a lower number then count some more for a very long time. I clocked the counter at over 3 minutes (I got past the time out issues) there are no PHP warnings.
Now the interesting part. If I insert an echo "x"; statement into the read loop, the script prints out the list of x's on the screen, the counters work as expected and time drops down to about 30 seconds. Same file and no other changes to the script.
It looks like the script is being swapped in and out with the variables being reset with random data. I'm not doing anything with memory management. It also does something else strange. If, while the script is running, I make a change to the script, the new script is sometimes executed (I added an echo statement that would run at the end, and it was executed)
Does anyone have a clue why this is happening?
I wrote a short script to simply count the records with fgetcsv, which is what the load uses. I run the script thru a browser. When I read anywhere 15,000 to 30,000 records, the counters that are being displayed go crazy, they go from 30, 000 to 700, then continue counting then they jump to 32,000, count some more then drop back to a lower number then count some more for a very long time. I clocked the counter at over 3 minutes (I got past the time out issues) there are no PHP warnings.
Now the interesting part. If I insert an echo "x"; statement into the read loop, the script prints out the list of x's on the screen, the counters work as expected and time drops down to about 30 seconds. Same file and no other changes to the script.
It looks like the script is being swapped in and out with the variables being reset with random data. I'm not doing anything with memory management. It also does something else strange. If, while the script is running, I make a change to the script, the new script is sometimes executed (I added an echo statement that would run at the end, and it was executed)
Does anyone have a clue why this is happening?
Last edited by mazzitech on Wed Dec 14, 2011 7:41 pm, edited 1 time in total.
-
mikeashfield
- Forum Contributor
- Posts: 159
- Joined: Sat Oct 22, 2011 10:50 am
-
mikeashfield
- Forum Contributor
- Posts: 159
- Joined: Sat Oct 22, 2011 10:50 am
Re: Long running script freaks out
Failing that, do something similar to this:
Code: Select all
<?php
x = 1;
for (i = 1; i < 100; i++) {
y = x;
echo x;
x = y++;
}
?>Re: Long running script freaks out
It might be easier for us to diagnose the problem if we could see the script.mazzitech wrote:Does anyone have a clue why this is happening?
Re: Long running script freaks out
The counting program did a few more things, such as counting (cnt++;) and it was writing a count record to a file (fopen, fwrite, fclose) on every iteration. I had another script reading that fie every 10 secs that displayed the counts. I had no need for the echo "x"; Changing the character from an X to a blank had no effect, the script still ran correctly. I tried putting that into the load script where the problem first surfaced and it still doesn't work right.
Re: Long running script freaks out
Here is the script
Code: Select all
<?php
error_reporting(E_ALL);
ini_set("max_execution_time", "3000");
ini_set('memory_limit','640M');
set_time_limit(2500); // set the max time for the script, 30 secs = 80,000 records
$cnt=0;
$cntin=0;
$loop=0;
$sw=0;
$c=0;
$maxlen=0;
$fhcnt = fopen("readcount.log", 'w');
fclose($fhcnt);
$fhin = fopen("selected_npidata_20111114.csv", 'r') or die("Failed to open Input file"); // NPI INput file
// echo $fhin;
$header = fgetcsv($fhin, 10000, ","); // Read the header for CB
while (($data = fgets($fhin, 10000)) !== FALSE) // read and convert the NPI data for CB
{
$cntin++;
$c++;
$hold = strlen($data);
if ($hold > $maxlen) {
echo "<p>".$hold."</p>";
$maxlen = $hold;
}
// echo " ";
$fhcnt = fopen("readcount.log", 'r+');
fwrite($fhcnt,"Record number ".$cntin." Max Length: ".$maxlen."<br />");
fclose($fhcnt);
if (file_exists("stop.log")) {
break;
}
}
fclose($fhin);
$fhcnt = fopen("readcount.log", 'r+');
fwrite($fhcnt,"Finished - Counted ".$cntin." records, Max Length".$maxlen);
fclose($fhcnt);
echo "<p>Finished!, Counted $cntin records Max Length: $maxlen</p>";
Last edited by Benjamin on Fri Dec 23, 2011 2:18 am, edited 1 time in total.
Reason: Added [syntax=php||htm||css||javascript||sql||etc] - Please use [syntax] tags when posting code in the forums! Thanks.
Reason: Added [syntax=php||htm||css||javascript||sql||etc] - Please use [syntax] tags when posting code in the forums! Thanks.
Re: Long running script freaks out
Solved! The culprit here is the load balancing scheme that the host is using. When the echo was used, the "sticky sessions" capability came into play. Without that, the load balancer didn't work properly. In order to get enough execution time, the script needs to be run by cron
Re: Long running script freaks out
Interesting. Thanks for sharing the solution.