Page 1 of 1

Make Copies of Websites

Posted: Mon Mar 30, 2009 11:45 pm
by kaje
OK, so I have a few pages that parse data from a 3rd party website everytime the page is loaded. I am wanting to prevent this and set up my website a way so that it doesn't parse the data from the 3rd party site everytime the page is opened but maybe only a couple times a day (to update the data). To do this, I am wanting to make a copy of the 3rd party page, save it onto my site, then allow my other page to parse data from the copy everytime it is opened.

So I have thought up the current method:

Code: Select all

$contents = file_get_html('http://www.site.com/');
file_put_contents('test.html', $contents);
$contents = file_get_html('http://www.site.com/2');
file_put_contents('test2.html', $contents);
$contents = file_get_html('http://www.site.com/3');
file_put_contents('test3.html', $contents);
$contents = file_get_html('http://www.site.com/4');
file_put_contents('test4.html', $contents);
However, after putting too many sites in here, I get the error:

Code: Select all

Fatal error: Allowed memory size of 18874368 bytes exhausted (tried to allocate 40 bytes) in /etc/etc/etc/simple_html_dom.php on line 618
So, what do I need to do to keep from taking up all the memory or should I be using a different method for doing this altogether?

Re: Make Copies of Websites

Posted: Tue Mar 31, 2009 12:07 am
by atonalpanic
Why don't you just use a cron job or windows task scheduler? Also check the docs at php.net to see if you need to flush the buffer or anything.

Re: Make Copies of Websites

Posted: Tue Mar 31, 2009 8:50 am
by kaje
atonalpanic wrote:Why don't you just use a cron job or windows task scheduler? Also check the docs at php.net to see if you need to flush the buffer or anything.
Well that's what I thought I was going to do with this? I was going to have a cron job run this php script a couple times a day. But, unfortunately, my host doesn't offer cronjob in the Control Panel so I haven't been inside of cronjobs to look at how I'm going to do it. It was just a recommendation by a friend to use the cron job on his server to run this script.

I've tried searching for this problem and keep running into the solution of "increasing the memory size." Considering I'm getting this error after 5 sites and I'm going to be copying 100+, I imagine my process of doing this isn't right as I think it's a little extreme to increase the memory size x20 or however much more when I add news sites.

Re: Make Copies of Websites

Posted: Wed Apr 01, 2009 3:19 pm
by kaje
Any ideas? Am I going about this the wrong way?

Re: Make Copies of Websites

Posted: Wed Apr 01, 2009 6:43 pm
by McInfo
I don't know much about file_get_html() because it isn't in the PHP manual. Have you tried using file_get_contents() instead?

Edit: This post was recovered from search engine cache.

Re: Make Copies of Websites

Posted: Wed Apr 01, 2009 10:12 pm
by kaje
McInfo wrote:I don't know much about file_get_html() because it isn't in the PHP manual. Have you tried using file_get_contents() instead?
That did the trick! Thanks!