Any ideas? Thanks in advance!Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 42 bytes) in C:\xampp\htdocs\simplehtmldom\simple_html_dom.php on line 618
PHP/Mysql screen scraper memory issue
Moderator: General Moderators
PHP/Mysql screen scraper memory issue
I have a screen scraper that I've written in php that takes info from a website and puts it into a mysql database. The problem I'm having is memory related... it's a very large site I'm attempting to scrape and after I'm probably 1/20th of the way done I get this error:
-
sammonster
- Forum Newbie
- Posts: 6
- Joined: Tue Aug 31, 2010 10:51 pm
Re: PHP/Mysql screen scraper memory issue
Sounds like the memory you have set in PHP (probably 128mb) is being exceeded from the amount of data you are trying to load. The only solution would be to use a terminal application or up the memory limit in the php.ini...
Re: PHP/Mysql screen scraper memory issue
Ok, but because I'm only able to scrape approximately 1/20th of the data would upping the data limit in the php.ini file be a possible solution to this issue?sammonster wrote:Sounds like the memory you have set in PHP (probably 128mb) is being exceeded from the amount of data you are trying to load. The only solution would be to use a terminal application or up the memory limit in the php.ini...
Not sure what you mean by "a terminal application"? Are you suggesting to break it down into manageable chunks and do one section at a time?
What I don't understand is why it uses progressively more and more memory as it goes through the links as I have the loop set to do one link at a time -- connect to the db, input the data, disconnect from the db, do it again. What is being stored in memory that is causing it to overload?
Thanks for your help!
-
sammonster
- Forum Newbie
- Posts: 6
- Joined: Tue Aug 31, 2010 10:51 pm
Re: PHP/Mysql screen scraper memory issue
it's mainly because you're loading the entire site in memory (I assume you are doing like a file_get_contents or curl request). Either way, the entire site's data is going to store inside of a block. Then, you probably do some sort of parse, and maybe even splitting it out into array chunks. I would do a memory dump through your application and see where the majority of your memory is getting blasted from.
The other thing is you could be getting into an infinite loop. But that would probably cause a timeout instead....
I dunno. As for a terminal application, I'm talking like split or AWK, or some other reading tool.
Upping your memory limit would probably help, but doing so is kind of bad practice unless you know for sure it's the amount of memory you're trying to load and not your code. Try posting some code and the url you're trying to read and maybe we can figure it out.
The other thing is you could be getting into an infinite loop. But that would probably cause a timeout instead....
I dunno. As for a terminal application, I'm talking like split or AWK, or some other reading tool.
Upping your memory limit would probably help, but doing so is kind of bad practice unless you know for sure it's the amount of memory you're trying to load and not your code. Try posting some code and the url you're trying to read and maybe we can figure it out.
Re: PHP/Mysql screen scraper memory issue
Unsetting all of the variables at the end of my while loop did the trick... thanks for your help!