this might be an fread memory issue

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
rhecker
Forum Contributor
Posts: 178
Joined: Fri Jul 11, 2008 5:49 pm

mysql memory issue, I think

Post by rhecker »

I am building a set of static html pages from a mysql table, but the content in some of the longer pages ends before they should. If I remove a little data before the ending location, then it extends the same length. This is on my local Win64 system (apache2, Mysql 5.3, PHP). fread is set to 16384 bytes. It seams like I am up against a memory limit somewhere, but I can't figure it out.

any thoughts?
rhecker
Forum Contributor
Posts: 178
Joined: Fri Jul 11, 2008 5:49 pm

this might be an fread memory issue

Post by rhecker »

I am building pages for a static website using fread set to 8192 bytes. None of these pages are very large, but on the larger ones, the content will cut off. I have been playing with memory variables in my.ini and php.ini but can't change the situation. This is just running on my windows 7 64 machine, apache 2, mysql, php 5.3 Any suggestions?

Code: Select all

while ($pagelist = mysql_fetch_array($page_set)) {
  $page_id = $pagelist['page_id'];
  echo "$page_id ";
  $srcurl = "http://bahamas/create_set/course_page.php?page_id=$page_id";
  $tempfilename = 'temp.html';
  $targetfilename="../html_content_files/$page_id.html";
@unlink($tempfilename);
$dynpage=fopen($srcurl, 'r');
if (!$dynpage){
	exit("no can do");
}
$htmldata=fread($dynpage, 8192);
fclose($dynpage);
$tempfile=fopen($tempfilename, 'w');
if (!$tempfile){
	exit("unable to open temp file for writing");
}
fwrite($tempfile, $htmldata);
fclose($tempfile);
$ok = copy ($tempfilename, $targetfilename);
unlink($tempfilename);
}
User avatar
Eran
DevNet Master
Posts: 3549
Joined: Fri Jan 18, 2008 12:36 am
Location: Israel, ME

Re: mysql memory issue, I think

Post by Eran »

It's probably a column type issue - you are using a 'TEXT' column type which can fit up to 64kb of text. You should change it to MEDIUMTEXT (16mb) or LONGTEXT (4gb)
rhecker
Forum Contributor
Posts: 178
Joined: Fri Jul 11, 2008 5:49 pm

Re: mysql memory issue, I think

Post by rhecker »

Good thought, but that's not it. There are about ten fields. If the content ends on field eight, and I then go back and delete the contents of field 3, field eight will then continue, and the pagen will then end in maybe the middle of field ten.

I am now thinking that this isn't a database issue, but a php issue, even though I have played with the memory allotment for fread, to no avail.
User avatar
Eran
DevNet Master
Posts: 3549
Joined: Fri Jan 18, 2008 12:36 am
Location: Israel, ME

Re: mysql memory issue, I think

Post by Eran »

Good thought, but that's not it.
Did you try it?

and please post some code and the error message you are receiving
rhecker
Forum Contributor
Posts: 178
Joined: Fri Jul 11, 2008 5:49 pm

Re: mysql memory issue, I think

Post by rhecker »

None of these text fields have more than three paragraphs of text, although many of them contain html, which I handle with htmlenties going in and html_entities_decode going out. But at your suggestion I tried changing text to medium text, with no change. However, I also commented out some of the html_entities_decode script, thinking it would free up resources and give me more content, but instead it gave me less. Unexpected!

Here is the code

Code: Select all

$page_set = mysql_query("SELECT page_id, title from courses WHERE program = 'course'");
while ($pagelist = mysql_fetch_array($page_set)) {
  $page_id = $pagelist['page_id'];
  echo "$page_id ";
  $srcurl = "http://bahamas/create_set/course_page.php?page_id=$page_id";
  $tempfilename = 'temp.html';
  $targetfilename="../html_content_files/$page_id.html";
@unlink($tempfilename);
$dynpage=fopen($srcurl, 'r');
if (!$dynpage){
	exit("no can do");
}
$htmldata=fread($dynpage, 8192);
fclose($dynpage);
$tempfile=fopen($tempfilename, 'w');
if (!$tempfile){
	exit("unable to open temp file for writing");
}
fwrite($tempfile, $htmldata);
fclose($tempfile);
$ok = copy ($tempfilename, $targetfilename);
unlink($tempfilename);
}
rhecker
Forum Contributor
Posts: 178
Joined: Fri Jul 11, 2008 5:49 pm

Re: mysql memory issue, I think

Post by rhecker »

Problem solved. When I added sleep(.1) to my loop, it gave the script time to finish writing the page.
rhecker
Forum Contributor
Posts: 178
Joined: Fri Jul 11, 2008 5:49 pm

Re: this might be an fread memory issue

Post by rhecker »

Problem solved.

When I added sleep(.1) to my loop, the script had time to finish writing the page.
User avatar
Weirdan
Moderator
Posts: 5978
Joined: Mon Nov 03, 2003 6:13 pm
Location: Odessa, Ukraine

Re: this might be an fread memory issue

Post by Weirdan »

If those files that got truncated were larger than 8KB that wasn't the problem with fread(), but rather how you use it. Most likely you want to use file_get_contents() instead.

Also it's not clear why would you first write to a temporary file and than copy it. While there's valid reasons to not write data directly to it's final destination (to avoid users getting partially written file for example) all scenarios I can think of require moving the file from its temporary location to the target directory.
rhecker
Forum Contributor
Posts: 178
Joined: Fri Jul 11, 2008 5:49 pm

Re: this might be an fread memory issue

Post by rhecker »

Thanks. Turns out sleep did not solve the issue. I will try your suggestion.
User avatar
Weirdan
Moderator
Posts: 5978
Joined: Mon Nov 03, 2003 6:13 pm
Location: Odessa, Ukraine

Re: this might be an fread memory issue

Post by Weirdan »

:arrow: Threads merged
rhecker
Forum Contributor
Posts: 178
Joined: Fri Jul 11, 2008 5:49 pm

Re: this might be an fread memory issue

Post by rhecker »

Weirdan, thanks, you were right. file_get_contents solved the problem.

Sorry for posting in two forums. At first I thought this was a database issue, then realized it wasn't
Post Reply