Page 1 of 1

Odd Memory problem

Posted: Fri Dec 02, 2005 11:49 am
by CantonDog
I'm not new to PHP.. new to XML.. We're now getting an xml feed and I was tasked to make the data 'look pretty' on the site. After searching and reading dozzens of xml tuts, finally came across an xml to array script. Basically turns the xml file into an array that I can work with. OK. Cool.
Did it.. it was working.. all was well with the universe.

Suddenly it stops working.. I examine the xml file they're sending and it went from 1.2MB to over 2MB... found out that I was excedding the memory limit (or at least that's the only error I could get generated for me).

Well, since I only needed a partial part of the xml file, I was able to write a script that only pulled the lines I needed. So now the 'new' xml file (2,091 lines) is significantly smaller than the 'original' one (33,000+ lines)... however, I'm still getting the SAME memory error... on a smaller file..

Anyone know how this could be? I even tried the 'Jerry-rig way' of setting ini_set memory limit on the script.. but no good..

The script is kinda long but what the heck.. I'll paste it in here. Hopefully someone can shed some light on this for me.

Code: Select all

<?php
$code = "-- bunch of javascript - took it out to save space in the forum here -- ";

function GetXMLTree ($xmldata)
{
	// we want to know if an error occurs
	ini_set ('track_errors', '1');

	$xmlreaderror = false;

	$parser = xml_parser_create ('ISO-8859-1');
	xml_parser_set_option ($parser, XML_OPTION_SKIP_WHITE, 1);
	xml_parser_set_option ($parser, XML_OPTION_CASE_FOLDING, 0);
	
        /// comented these lines out -- that's how I got the "Allowed memory size of xxxxxx bytes exhausted" error to show up
        //if (!xml_parse_into_struct ($parser, $xmldata, $vals, $index)) {
	//	$xmlreaderror = true;
	//	echo "error";
	//}
	xml_parser_free ($parser);

	if (!$xmlreaderror) {
		$result = array ();
		$i = 0;
		if (isset ($vals [$i]['attributes']))
			foreach (array_keys ($vals [$i]['attributes']) as $attkey)
			$attributes [$attkey] = $vals [$i]['attributes'][$attkey];

		$result [$vals [$i]['tag']] = array_merge ($attributes, GetChildren ($vals, $i, 'open'));
		//$result [$vals [$i][’tag’]] = array_merge ($attributes, GetChildren ($vals, $i, $vals[$i][’type’]));
	}

	ini_set ('track_errors', '0');
	return $result;
}

function GetChildren ($vals, &$i, $type)
{
	if ($type == 'complete') {
		if (isset ($vals [$i]['value']))
			return ($vals [$i]['value']);
		else
			return '';
	}

	$children = array (); // Contains node data

	/* Loop through children */
	while ($vals [++$i]['type'] != 'close') {
		$type = $vals [$i]['type'];
		// first check if we already have one and need to create an array
		if (isset ($children [$vals [$i]['tag']])) {
			if (is_array ($children [$vals [$i]['tag']])) {
				$temp = array_keys ($children [$vals [$i]['tag']]);
				// there is one of these things already and it is itself an array
				if (is_string ($temp [0])) {
					$a = $children [$vals [$i]['tag']];
					unset ($children [$vals [$i]['tag']]);
					$children [$vals [$i]['tag']][0] = $a;
				}
			} else {
				$a = $children [$vals [$i]['tag']];
				unset ($children [$vals [$i]['tag']]);
				$children [$vals [$i]['tag']][0] = $a;
			}

			$children [$vals [$i]['tag']][] = GetChildren ($vals, $i, $type);
		} else{
			$children [$vals [$i]['tag']] = GetChildren ($vals, $i, $type);
		}
		// I don't think I need attributes but this is how I would do them:
		if (isset ($vals [$i]['attributes'])) {
			$attributes = array ();
			foreach (array_keys ($vals [$i]['attributes']) as $attkey)
			$attributes [$attkey] = $vals [$i]['attributes'][$attkey];
			// now check: do we already have an array or a value?
			if (isset ($children [$vals [$i]['tag']])) {
				// case where there is an attribute but no value, a complete with an attribute in other words
				if ($children [$vals [$i]['tag']] == '') {
					unset ($children [$vals [$i]['tag']]);
					$children [$vals [$i]['tag']] = $attributes;
				}
				// case where there is an array of identical items with attributes
				elseif (is_array ($children [$vals [$i]['tag']])) {
					$index = count ($children [$vals [$i]['tag']]) - 1;
					// probably also have to check here whether the individual item is also an array or not or what... all a bit messy
					if ($children [$vals [$i]['tag']][$index] == '') {
						unset ($children [$vals [$i]['tag']][$index]);
						$children [$vals [$i]['tag']][$index] = $attributes;
					}
					$children [$vals [$i]['tag']][$index] = array_merge ($children [$vals [$i]['tag']][$index], $attributes);
				} else {
					$value = $children [$vals [$i]['tag']];
					unset ($children [$vals [$i]['tag']]);
					$children [$vals [$i]['tag']]['value'] = $value;
					$children [$vals [$i]['tag']] = array_merge ($children [$vals [$i]['tag']], $attributes);
				}
			} else{
				$children [$vals [$i]['tag']] = $attributes;
			}
		}
	}

	return $children;
}
$url = "ski_reports_inc3.html"; //URL of the XML FEED
$contents = file_get_contents($url);

$data = GetXMLTree ($contents);

//print_r($data);

///////  If I uncomment the above line, and delete everything below the script WORKS but I need the info below to make the data look 'pretty'

/////////////////// output all data //////////////////////////

$Resorts = count($data["State"][1]["SkiArea"]);

///print quick links for resorts (divide by 2 for 2 columns)
$ResortsHalf = $Resorts/2;
$ResortsHalf = ceil($ResortsHalf);

$code .= "<table border=\"0\" cellpadding=\"2\" cellspacing=\"0\">
		<tr colspan='2'><td>Click a resort name below to jump to that resort</td></tr><tr><td>";

for($x=0;$x<$Resorts;$x++){
	$code .= "<li><a href=\"#".$data["State"][1]["SkiArea"][$x]["Name"]."\">".$data["State"][1]["SkiArea"][$x]["Name"]."</a></li>";
	if($x == $ResortsHalf-1){
		$code .= "</td><td>";
	}
}

$code .= "</td></tr></table><p>&nbsp;</p>";

for($i=0;$i<$Resorts;$i++){
	$code .= "<table width=\"510\" border=\"1\" cellpadding=\"2\" cellspacing=\"0\" bordercolor=\"#990000\">
  <tr>
    <td bgcolor=\"#990000\"><a name=\"".$data["State"][1]["SkiArea"][$i]["Name"]."\"><span class=\"skiteamResortName\">".$data["State"][1]["SkiArea"][$i]["Name"]."<br>
    </span><span class=\"style7\"><span class=\"skiteamTinyTextWhite\">Updated: ".$data["State"][1]["SkiArea"][$i]["LastUpdate"]."</span></span>
   </td>
   <td bgcolor=\"#990000\" align=\"right\">";
   
   $MNCLinks = count($data["State"][1]["SkiArea"][$i]["MNCLink"]);
   for($Pointer=0;$Pointer<$MNCLinks;$Pointer++){
   	if($data["State"][1]["SkiArea"][$i]["MNCLink"][$Pointer]["label"] == "5-Day Forecast"){
   		if($data["State"][1]["SkiArea"][$i]["MNCLink"][$Pointer]["value"] == ""){
   			$code .= "<a href=\"".$data["State"][1]["SkiArea"][$i]["MNCLink"][$Pointer][0]."\" target=\"blank\"><span class=\"skiteamTinyTextWhite\">".$data["State"][1]["SkiArea"][$i]["MNCLink"][$Pointer]["label"]."</span></a>";
   		}else{
   			$code .= "<a href=\"".$data["State"][1]["SkiArea"][$i]["MNCLink"][$Pointer]["value"]."\" target=\"blank\"><span class=\"skiteamTinyTextWhite\">".$data["State"][1]["SkiArea"][$i]["MNCLink"][$Pointer]["label"]."</span></a>";
   		}
   	}
   	
   	if($data["State"][1]["SkiArea"][$i]["MNCLink"][$Pointer]["label"] == "Snow Report"){
   		$code .= "<br><a href=\"".$data["State"][1]["SkiArea"][$i]["MNCLink"][$Pointer][0]."\" target=\"blank\"><span class=\"skiteamTinyTextWhite\">".$data["State"][1]["SkiArea"][$i]["MNCLink"][$Pointer]["label"]."</span></a>";
   	}
   	
   }
$code .= "&nbsp;</td>
  </tr>
  <tr>
    <td colspan=\"2\" bgcolor=\"#999999\"><div align=\"center\" class=\"skiteamHeaderWhite\"><strong>Snow Info </strong></div></td>
  </tr>
  <tr>
    <td colspan=\"2\"><table width=\"100%\" border=\"0\" align=\"center\" cellpadding=\"0\" cellspacing=\"3\">
      <tr>
        <td><div align=\"center\" class=\"skiteamTinyText\">New 24 </div></td>
        <td><div align=\"center\" class=\"skiteamTinyText\">New 48 </div></td>
        <td><div align=\"center\" class=\"skiteamTinyText\">New 72 </div></td>
        <td><div align=\"center\" class=\"skiteamTinyText\">Base Depth </div></td>
        <td><div align=\"center\" class=\"skiteamTinyText\">Top Depth </div></td>
        <td><div align=\"center\" class=\"skiteamTinyText\">Condition</div></td>
      </tr>
      <tr>
        <td><div align=\"center\" class=\"skiteamTinyText\">".$data["State"][1]["SkiArea"][$i]["NewSnow24"][0]["value"]."\"</div></td>
        <td><div align=\"center\" class=\"skiteamTinyText\">".$data["State"][1]["SkiArea"][$i]["NewSnow48"][0]["value"]."\"</div></td>
        <td><div align=\"center\" class=\"skiteamTinyText\">".$data["State"][1]["SkiArea"][$i]["NewSnow72"][0]["value"]."\"</div></td>
        <td><div align=\"center\" class=\"skiteamTinyText\">".$data["State"][1]["SkiArea"][$i]["BaseDepth"][0]["value"]."\"</div></td>
        <td><div align=\"center\" class=\"skiteamTinyText\">".$data["State"][1]["SkiArea"][$i]["TopDepth"][0]["value"]."\"</div></td>
        <td><div align=\"center\" class=\"skiteamTinyText\">";
        $ConditionsCount = count($data["State"][1]["SkiArea"][$i]["Condition"]);
        for($j=0;$j<$ConditionsCount;$j++){
        	if($j > 0){
        		$code .= "<br>";
        	}
        	$code .= $data["State"][1]["SkiArea"][$i]["Condition"][$j]["Name"];
        }
        $code .= "</div></td>
      </tr>
    </table></td>
  </tr>
  <tr>
    <td colspan=\"2\" bgcolor=\"#999999\"><div align=\"center\" class=\"skiteamHeaderWhite\"><strong>Lift Info </strong></div></td>
  </tr>
  <tr>
    <td colspan=\"2\"><table width=\"100%\" border=\"0\" align=\"center\" cellpadding=\"0\" cellspacing=\"3\">
      <tr class=\"skiteamTinyText\">
        <td><div align=\"center\" class=\"skiteamTinyText\">Lifts Open </div></td>
        <td><div align=\"center\" class=\"skiteamTinyText\">Total Lifts </div></td>
        <td><div align=\"center\" class=\"skiteamTinyText\">% Open </div></td>
        <td><div align=\"center\" class=\"skiteamTinyText\">Number of Runs </div></td>
        <td><div align=\"center\" class=\"skiteamTinyText\">Acres Open </div></td>
      </tr>
      <tr class=\"skiteamTinyText\">
        <td><div align=\"center\" class=\"skiteamTinyText\">".$data["State"][1]["SkiArea"][$i]["NumLiftsOpen"]."</div></td>
        <td><div align=\"center\" class=\"skiteamTinyText\">".$data["State"][1]["SkiArea"][$i]["NumLiftsTotal"]."</div></td>
        <td><div align=\"center\" class=\"skiteamTinyText\">".$data["State"][1]["SkiArea"][$i]["PerLiftsOpen"]."%</div></td>
        <td><div align=\"center\" class=\"skiteamTinyText\">".$data["State"][1]["SkiArea"][$i]["NumberOfRuns"]."</div></td>
        <td><div align=\"center\" class=\"skiteamTinyText\">".$data["State"][1]["SkiArea"][$i]["AcresOpen"]."</div></td>
      </tr>
    </table></td>
  </tr>
  <tr>
    <td colspan=\"2\" bgcolor=\"#CCCCCC\"><p class=\"skiteamTinyText\">";
    if($data["State"][1]["SkiArea"][$i]["Hours"] != ""){
    	$code .= "Hours: ".$data["State"][1]["SkiArea"][$i]["Hours"]."<br>";
    }
    if($data["State"][1]["SkiArea"][$i]["URL"] != ""){
    	$code .= "<a href=\"".$data["State"][1]["SkiArea"][$i]["URL"]."\" target=\"blank\">Website Link</a><br>";
    }
    if($data["State"][1]["SkiArea"][$i]["Email"] != ""){
    	$code .= "E-mail: ".$data["State"][1]["SkiArea"][$i]["Email"]."<br>";
    }
    if($data["State"][1]["SkiArea"][$i]["CallAheadPhone"] != ""){
    	$code .= "Phone: ".$data["State"][1]["SkiArea"][$i]["CallAheadPhone"];
    }
    
    $code .="</p></td>
  </tr>
</table><p>&nbsp;</p>";

}


unset($data);
echo $code;
$tmpfile = fopen("ski_reports_inc.html","w+"); 
$fp = fwrite($tmpfile,$code); 
fclose($tmpfile); 
flush (); 

unset($code);

?>
thanks in advance for your time with my question.

Posted: Fri Dec 02, 2005 11:54 am
by neophyte
Memory limits can be set in a .htaccess file. Have you tried that?

I did a quick google I found this:

http://www.ploghost.com/forums/lofivers ... /t406.html

Posted: Fri Dec 02, 2005 12:01 pm
by CantonDog
neophyte,

thanks.. I tried that just now.. wasn't sure if that method is different than setting the memory limit in the file itself but I figured I'd give it a shot. Same result though.. no-go.. any other ideas?

Posted: Fri Dec 02, 2005 12:16 pm
by trukfixer
do you have the xml_rpc extension? if not, can you get it installed? it might come in handy for ya.. I use it quite a bit at work, for intranetwork machine communications (a central control machine sends out data in a secure stream to the other machines on a specific port, and at the other end, the xml is parsed back into data passed to teh python or perl scripts that do the actual work)

It works beautifully for mee, even with large data files that otherwise would cause php to exceed memory limits

http://us2.php.net/manual/en/ref.xmlrpc.php

Posted: Fri Dec 02, 2005 1:03 pm
by CantonDog
trukfixer - not sure if it is or not.. however, if it is.. not sure that would help me out with this issue.

Posted: Fri Dec 02, 2005 3:22 pm
by pickle
You're still reading in the entire contents of the file:

Code: Select all

$contents = file_get_contents($url);
You may not be able to call ini_set_memory_limit() from your script - it could be a restricted function. Do you have access to php.ini? I'd recommend changing the memory limit there.

However, this is only a bandaid - you could conceivably (sp?) reach any memory limit you set. Why not read it in the file the old fashioned way? Open a file handle and read in one line at a time? That's about the only way I know of restricting what you pull out of a file, without first reading in the whole file.

Posted: Fri Dec 02, 2005 3:56 pm
by CantonDog
you're right.. however, the entire contents of the file

Code: Select all

$url = "ski_reports_inc3.html";
is the smaller version. There is a different script that parses the original file and creates this smaller one. So in this case, reading the whole file should be ok.

Posted: Fri Dec 02, 2005 4:40 pm
by Weirdan
$code string could become very large, you might want to echo/fwrite directly instead of accumulating the entire output.