Page 1 of 1

PHP Allowed memory size problem.

Posted: Tue Oct 27, 2009 8:47 am
by joblafors
Hello.
I`m writing web crawler script for one specific site, and i`m using simple_html_dom to find links and specific information in site.
But I stumbled on PHP memory problem

Code: Select all

Fatal error: Allowed memory size of 10485760 bytes exhausted (tried to allocate 16 bytes) in *****\simple_html_dom.php on line 924
.
I changed memory limit to 30MB, 50MB .. but this doesn't resolve the problem because eventually memory will be full, that is if script crawls few pages deeper in site.
Are there any suggestions how to manage memory in PHP, or are there some better html dom scripts to use (with swap files or smth like that)
Thank you !!

Re: PHP Allowed memory size problem.

Posted: Tue Oct 27, 2009 11:48 am
by infolock
I would start doing something like this on every 20 lines of code...

Code: Select all

 
echo "<br />" . __LINE__ . "<br />";
exit;
 

I would step through and remove the exit from a line, refresh the page, remove another exit, refresh, etc until you get to the line where it's crashing.

This usually happens when you have an infinite loop, a request is failing, or a file is too big to place into memory.


More than likely this is due to a loop issue though...



Hope this helps