Page 1 of 1

Memory Size exausted

Posted: Sun Oct 10, 2004 9:12 pm
by tstimple
Hello to all,

I have a php page that I want to use to manipulate a txt file.
The code reads the entire text file (CSV) into an array then makes changes to several of the data fields by querying a mysql database.

The code works correctly when I have been testing with a sample txt file that has only a few dozen lines.
When I try to run it using the actual text file that has ~5000 lines I get the following meesage:

"Fatal error: Allowed memory size of 18388608 bytes exhausted (tried to allocate 12 bytes)"


Here is my code:

Code: Select all

<?php
mysql_connect("CONNECTION","USER","PASS"); 
mysql_select_db("DATABASE");

chdir('datafeed');

$textfile="VEHICLES.TXT";
$newfile="NEWVEHICLES.TXT";

// put the entire csv file into an array 
$file_name = "VEHICLES.TXT"; 

$all_lines = file($file_name); 

// count the number of lines in the file 
// and store it in a variable 

$how_many_lines = count($all_lines); 

// take each line and 
//  1. separate each line into its individual elements 
//      (aka fields aka pieces of information) using 
//      the EXPLODE () function. 
//  2. grab the value(s) you are going to change 
//      and place them in a variable(s). Remember, 
//      the count for the elements starts at 0 (zero). 
//      Also be careful that your variables don't 
//      contain extraneous parts (like extra spaces, 
//      newlines and tabs). Use the TRIM() function to 
//      clean up. 
//  3. do your change. 


// create a holding variable for the new contents 
$contents = ""; 

for ($i = 0; $i < $how_many_lines; $i++) { 

// separate each line into elements 
  $all_fields[$i] = explode(',', $all_lines[$i]); 

// clean up each element 
  $PSAMID = trim($all_fields[$i][0]); 
  $VIN = trim($all_fields[$i][1]); 
  $Stock = trim($all_fields[$i][2]); 
  $Status = trim($all_fields[$i][3]);
  $Make = trim($all_fields[$i][4]);
  	if($Make <> "A_"){
		$Make=$Make;
		}
	else{
		$Make="PO";
		}	
  $Model = trim($all_fields[$i][5]);
  $Year = trim($all_fields[$i][6]);
  $Mileage = trim($all_fields[$i][7]);
  $BODY = trim($all_fields[$i][8]);
  $Engine = trim($all_fields[$i][9]);
  $Liter = trim($all_fields[$i][10]);
  $Induction = trim($all_fields[$i][11]);
  $Transmission = trim($all_fields[$i][12]);
  $Color = trim($all_fields[$i][13]);
  $Price = trim($all_fields[$i][14]);
  $Cost = trim($all_fields[$i][15]);
  $Warranty = trim($all_fields[$i][16]);
  $Limited = trim($all_fields[$i][17]);
  $WarrantyCde = trim($all_fields[$i][18]);
  $Parts = trim($all_fields[$i][21]);
  $Labor = trim($all_fields[$i][22]);
  $Months = trim($all_fields[$i][23]);
  $Milescovered = trim($all_fields[$i][24]);
  $unknown1 = trim($all_fields[$i][25]);
  $ManYear = trim($all_fields[$i][26]);
  $Xservice = trim($all_fields[$i][27]);
  $ListDate = trim($all_fields[$i][28]);
  $O1 = trim($all_fields[$i][30]);
  $O2 = trim($all_fields[$i][31]);
  $O3 = trim($all_fields[$i][32]);
  $O4 = trim($all_fields[$i][33]);
  $O5 = trim($all_fields[$i][34]);
  $O6 = trim($all_fields[$i][35]);
  $O7 = trim($all_fields[$i][36]);
  $O8 = trim($all_fields[$i][$i] [37]); 
  $O9 = trim($all_fields[$i][$i] [38]); 
  $O10 = trim($all_fields[$i][$i] [39]); 
  $O11 = trim($all_fields[$i][40]);
  $O12 = trim($all_fields[$i][41]);
  $O13 = trim($all_fields[$i][42]);
  $O14 = trim($all_fields[$i][43]);
  $O15 = trim($all_fields[$i][44]);
  $O16 = trim($all_fields[$i][45]);
  $O17 = trim($all_fields[$i][46]);
  $O18 = trim($all_fields[$i][47]);
  $O19 = trim($all_fields[$i][48]);
  $O20 = trim($all_fields[$i][49]);
  $O21 = trim($all_fields[$i][50]);
  $O22 = trim($all_fields[$i][51]);
  $O23 = trim($all_fields[$i][52]);
  $O24 = trim($all_fields[$i][53]);
  $O25 = trim($all_fields[$i][54]);
  $O26 = trim($all_fields[$i][55]);
  $O27 = trim($all_fields[$i][56]);
  $O28 = trim($all_fields[$i][57]);
  $O29 = trim($all_fields[$i][58]);
  $O30 = trim($all_fields[$i][59]);
  $O31 = trim($all_fields[$i][60]);
  $O32 = trim($all_fields[$i][61]);
  $O33 = trim($all_fields[$i][62]);
  $O34 = trim($all_fields[$i][63]);
  $O35 = trim($all_fields[$i][64]);
  $DealerCode=substr($PSAMID, 1, 5);
  $CountyQuery = mysql_query("SELECT * FROM DealerByCounty WHERE PSAMID='$DealerCode'") or die('Error getting County.<br>' . mysql_error());
 	$x=mysql_fetch_row($CountyQuery);
 	$County = $x[2];
  $CountyLU = "";
  $photocode = trim($all_fields[$i][67]);
  $BodyCode=substr($BODY, 1, 2);
  $BodySearch = mysql_query("SELECT * FROM CARPARTS WHERE PartCode='$BodyCode'") or die('Error getting Body Type.<br>' . mysql_error());
	$x=mysql_fetch_row($BodySearch);
	$BodyType = $x[2];
  $optioncodes = "" ;
  $RecNo = $i+1;
  


// reassemble the line 
  $line = $PSAMID . "," . $VIN . "," . $Stock . "," . $Status . "," . $Make . "," . $Model . "," . $Year . "," . $Mileage . "," . $BODY . "," . $Engine . "," . $Liter . "," . $Induction . "," . $Transmission . "," . $Color . "," . $Price . "," . $Cost . "," . $Warranty . "," . $Limited . "," . $WarrantyCode . "," . $Parts . "," . $Labor . "," . $Months . "," . $Milescovered . "," . $unknown1 . "," . $ManYear . "," . $Xservice . "," . $ListDate . "," . $O1 . "," . $O2 . "," . $O3 . "," . $O4 . "," . $O5 . "," . $O6 . "," . $O7 ."," . $O8 . "," . $O9 . "," . $O10 ."," . $O11 . "," . $O12 . "," . $O13 ."," . $O14 . "," . $O15 . "," . $O16 ."," . $O17 . "," . $O18 . "," . $O19 ."," . $O20 . "," . $O21 . "," . $O22 ."," . $O23 . "," . $O24 . "," . $O25 ."," . $O26 . "," . $O27 . "," . $O28 ."," . $O29 . "," . $O30 . "," . $O31 ."," . $O32 . "," . $O33 . "," . $O34 ."," . $O35 . "," . $County . "," . $CountyLU . "," . $photocode . "," . $BodyType . "," . $optioncodes . "," . $RecNo . "\n"; 

// add the line to the contents 
  $contents = $contents . $line; 
  } 

// open the csv file in the write mode 
$fp = fopen($newfile, "w"); 

// write the contents to the file 
fwrite($fp, $contents); 

//close the file 
fclose($fp);
?>
Is this a limitation of my (hosted) server?
Any suggestions???

PS - I found (and addapted) this code from a post I originally found on this forum.

Posted: Sun Oct 10, 2004 11:14 pm
by feyd
yes, it's a limitation by your host.

You can use [php_man]fgets[/php_man]() with a [php_man]fopen[/php_man]ed file to get around it. Also, make sure you use [php_man]mysql_free_result[/php_man] when you are done with each of those selects, or mysql will run outta memory. :)

Posted: Mon Oct 11, 2004 2:03 am
by mudkicker
maybe using file_put_contents instead fwrite,fopen etc. would beter.. for al $O13 = trim($all_fields[$i][42]); things you could do the "trim" with a loop ???

Posted: Mon Oct 11, 2004 2:04 am
by feyd
warning: [php_man]file_put_contents[/php_man]() is php5 only.

Posted: Mon Oct 11, 2004 2:11 am
by mudkicker
oops ;) i was usin 5 heheh :D
but what about the "loop" thing?
feyd, isn't it more optimized?

Posted: Mon Oct 11, 2004 2:27 am
by feyd
mudkicker wrote:feyd, isn't it more optimized?
isn't what more optimized?

file_put_contents is marginally optimal over fopen, fwrite, fclose. Mostly because that's exactly what file_put_contents does, but because those functions are called through C instead of PHP, they are slightly faster.

Posted: Mon Oct 11, 2004 2:38 am
by mudkicker
No no, i meant that :arrow:
mudkicker wrote:for all $O13 = trim($all_fields[$i][42]); things you could do the "trim" with a loop ???

:?: :roll:

Posted: Sun Oct 17, 2004 5:12 pm
by feyd
yes, all the trims could be done with a loop quite easily, however I'd use a preg_split to do the bulk of the trimming for me. :)

As for everything else: A loop in combination with an array of variable names to create could be used create all the variables tstimple is using at the moment.