Page 1 of 1

mysql memory issue, I think

Posted: Fri Aug 20, 2010 3:48 pm
by rhecker
I am building a set of static html pages from a mysql table, but the content in some of the longer pages ends before they should. If I remove a little data before the ending location, then it extends the same length. This is on my local Win64 system (apache2, Mysql 5.3, PHP). fread is set to 16384 bytes. It seams like I am up against a memory limit somewhere, but I can't figure it out.

any thoughts?

this might be an fread memory issue

Posted: Fri Aug 20, 2010 4:21 pm
by rhecker
I am building pages for a static website using fread set to 8192 bytes. None of these pages are very large, but on the larger ones, the content will cut off. I have been playing with memory variables in my.ini and php.ini but can't change the situation. This is just running on my windows 7 64 machine, apache 2, mysql, php 5.3 Any suggestions?

Code: Select all

while ($pagelist = mysql_fetch_array($page_set)) {
  $page_id = $pagelist['page_id'];
  echo "$page_id ";
  $srcurl = "http://bahamas/create_set/course_page.php?page_id=$page_id";
  $tempfilename = 'temp.html';
  $targetfilename="../html_content_files/$page_id.html";
@unlink($tempfilename);
$dynpage=fopen($srcurl, 'r');
if (!$dynpage){
	exit("no can do");
}
$htmldata=fread($dynpage, 8192);
fclose($dynpage);
$tempfile=fopen($tempfilename, 'w');
if (!$tempfile){
	exit("unable to open temp file for writing");
}
fwrite($tempfile, $htmldata);
fclose($tempfile);
$ok = copy ($tempfilename, $targetfilename);
unlink($tempfilename);
}

Re: mysql memory issue, I think

Posted: Fri Aug 20, 2010 4:24 pm
by Eran
It's probably a column type issue - you are using a 'TEXT' column type which can fit up to 64kb of text. You should change it to MEDIUMTEXT (16mb) or LONGTEXT (4gb)

Re: mysql memory issue, I think

Posted: Fri Aug 20, 2010 4:49 pm
by rhecker
Good thought, but that's not it. There are about ten fields. If the content ends on field eight, and I then go back and delete the contents of field 3, field eight will then continue, and the pagen will then end in maybe the middle of field ten.

I am now thinking that this isn't a database issue, but a php issue, even though I have played with the memory allotment for fread, to no avail.

Re: mysql memory issue, I think

Posted: Fri Aug 20, 2010 4:54 pm
by Eran
Good thought, but that's not it.
Did you try it?

and please post some code and the error message you are receiving

Re: mysql memory issue, I think

Posted: Fri Aug 20, 2010 5:12 pm
by rhecker
None of these text fields have more than three paragraphs of text, although many of them contain html, which I handle with htmlenties going in and html_entities_decode going out. But at your suggestion I tried changing text to medium text, with no change. However, I also commented out some of the html_entities_decode script, thinking it would free up resources and give me more content, but instead it gave me less. Unexpected!

Here is the code

Code: Select all

$page_set = mysql_query("SELECT page_id, title from courses WHERE program = 'course'");
while ($pagelist = mysql_fetch_array($page_set)) {
  $page_id = $pagelist['page_id'];
  echo "$page_id ";
  $srcurl = "http://bahamas/create_set/course_page.php?page_id=$page_id";
  $tempfilename = 'temp.html';
  $targetfilename="../html_content_files/$page_id.html";
@unlink($tempfilename);
$dynpage=fopen($srcurl, 'r');
if (!$dynpage){
	exit("no can do");
}
$htmldata=fread($dynpage, 8192);
fclose($dynpage);
$tempfile=fopen($tempfilename, 'w');
if (!$tempfile){
	exit("unable to open temp file for writing");
}
fwrite($tempfile, $htmldata);
fclose($tempfile);
$ok = copy ($tempfilename, $targetfilename);
unlink($tempfilename);
}

Re: mysql memory issue, I think

Posted: Fri Aug 20, 2010 5:29 pm
by rhecker
Problem solved. When I added sleep(.1) to my loop, it gave the script time to finish writing the page.

Re: this might be an fread memory issue

Posted: Fri Aug 20, 2010 5:32 pm
by rhecker
Problem solved.

When I added sleep(.1) to my loop, the script had time to finish writing the page.

Re: this might be an fread memory issue

Posted: Fri Aug 20, 2010 5:52 pm
by Weirdan
If those files that got truncated were larger than 8KB that wasn't the problem with fread(), but rather how you use it. Most likely you want to use file_get_contents() instead.

Also it's not clear why would you first write to a temporary file and than copy it. While there's valid reasons to not write data directly to it's final destination (to avoid users getting partially written file for example) all scenarios I can think of require moving the file from its temporary location to the target directory.

Re: this might be an fread memory issue

Posted: Fri Aug 20, 2010 5:55 pm
by rhecker
Thanks. Turns out sleep did not solve the issue. I will try your suggestion.

Re: this might be an fread memory issue

Posted: Fri Aug 20, 2010 5:55 pm
by Weirdan
:arrow: Threads merged

Re: this might be an fread memory issue

Posted: Fri Aug 20, 2010 6:11 pm
by rhecker
Weirdan, thanks, you were right. file_get_contents solved the problem.

Sorry for posting in two forums. At first I thought this was a database issue, then realized it wasn't