Make Copies of Websites

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
kaje
Forum Newbie
Posts: 11
Joined: Sat Feb 28, 2009 12:01 pm

Make Copies of Websites

Post by kaje »

OK, so I have a few pages that parse data from a 3rd party website everytime the page is loaded. I am wanting to prevent this and set up my website a way so that it doesn't parse the data from the 3rd party site everytime the page is opened but maybe only a couple times a day (to update the data). To do this, I am wanting to make a copy of the 3rd party page, save it onto my site, then allow my other page to parse data from the copy everytime it is opened.

So I have thought up the current method:

Code: Select all

$contents = file_get_html('http://www.site.com/');
file_put_contents('test.html', $contents);
$contents = file_get_html('http://www.site.com/2');
file_put_contents('test2.html', $contents);
$contents = file_get_html('http://www.site.com/3');
file_put_contents('test3.html', $contents);
$contents = file_get_html('http://www.site.com/4');
file_put_contents('test4.html', $contents);
However, after putting too many sites in here, I get the error:

Code: Select all

Fatal error: Allowed memory size of 18874368 bytes exhausted (tried to allocate 40 bytes) in /etc/etc/etc/simple_html_dom.php on line 618
So, what do I need to do to keep from taking up all the memory or should I be using a different method for doing this altogether?
atonalpanic
Forum Commoner
Posts: 29
Joined: Mon Mar 02, 2009 10:20 pm

Re: Make Copies of Websites

Post by atonalpanic »

Why don't you just use a cron job or windows task scheduler? Also check the docs at php.net to see if you need to flush the buffer or anything.
kaje
Forum Newbie
Posts: 11
Joined: Sat Feb 28, 2009 12:01 pm

Re: Make Copies of Websites

Post by kaje »

atonalpanic wrote:Why don't you just use a cron job or windows task scheduler? Also check the docs at php.net to see if you need to flush the buffer or anything.
Well that's what I thought I was going to do with this? I was going to have a cron job run this php script a couple times a day. But, unfortunately, my host doesn't offer cronjob in the Control Panel so I haven't been inside of cronjobs to look at how I'm going to do it. It was just a recommendation by a friend to use the cron job on his server to run this script.

I've tried searching for this problem and keep running into the solution of "increasing the memory size." Considering I'm getting this error after 5 sites and I'm going to be copying 100+, I imagine my process of doing this isn't right as I think it's a little extreme to increase the memory size x20 or however much more when I add news sites.
kaje
Forum Newbie
Posts: 11
Joined: Sat Feb 28, 2009 12:01 pm

Re: Make Copies of Websites

Post by kaje »

Any ideas? Am I going about this the wrong way?
User avatar
McInfo
DevNet Resident
Posts: 1532
Joined: Wed Apr 01, 2009 1:31 pm

Re: Make Copies of Websites

Post by McInfo »

I don't know much about file_get_html() because it isn't in the PHP manual. Have you tried using file_get_contents() instead?

Edit: This post was recovered from search engine cache.
Last edited by McInfo on Mon Jun 14, 2010 11:18 am, edited 1 time in total.
kaje
Forum Newbie
Posts: 11
Joined: Sat Feb 28, 2009 12:01 pm

Re: Make Copies of Websites

Post by kaje »

McInfo wrote:I don't know much about file_get_html() because it isn't in the PHP manual. Have you tried using file_get_contents() instead?
That did the trick! Thanks!
Post Reply