PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!
Moderator: General Moderators
rgasch
Forum Newbie
Posts: 3 Joined: Wed Sep 03, 2008 8:09 am
Post
by rgasch » Wed Sep 03, 2008 9:07 am
Hi,
I'm having a strange problem that I can't figure out. I'm trying to generate a large XML file using XMLWriter but the file I generate never grows beyond 3138986 bytes (whether I use the XMLWriter to write to a file/URI or generate the XML in memory and then pass it though gzwrite() doesn't affect the outcome; the (gunzipped) file never grows beyond 3138986 bytes.
Here's some code excepts of what I'm doing:
Code: Select all
$writer = new XMLWriter();
if ($this->_gzip) {
$writer->openMemory();
$fp = gzopen ($this->_filename.'.gz', 'wb');
} else {
$writer->openURI($this->_filename);
}
$writer->startDocument('1.0', 'ISO-8859-15');
$writer->setIndent(4);
$writer->startElement('Products');
$count = 0;
$page = 0;
$objArray = new PNProductArray ();
$colArray = getColumnsArray('table');
do {
$data = selectFromDB ('$page*$this->_pagesize, $this->_pagesize);
foreach ($data as $dat) {
$writer->startElement('Product');
foreach ($colArray as $field) {
$writer->writeElement($field, $dat[$field]);
}
$writer->endElement();
$count++;
}
$page++;
} while ($data);
$writer->endElement();
$writer->endDocument();
if ($this->_gzip) {
$xml = $writer->outputMemory();
$fp = gzopen ($this->_filename.'.gz', 'wb');
gzwrite ($fp, $xml);
gzclose ($fp);
} else {
$writer->flush();
}
In summary, no matter what I do, I can't seem to generate an XML file greater than 3138986 bytes. I've tried flushing the buffer after every DB fetch (if writing to a non-compressed file) but nothing helped. I'm not running into any (visible) PHP memory or processing time limits.
Any ideas would be much appreciated.
Thanks
Rob
andyhoneycutt
Forum Contributor
Posts: 468 Joined: Wed Aug 27, 2008 10:02 am
Location: Idaho Falls
Post
by andyhoneycutt » Wed Sep 03, 2008 9:42 am
can you post a tail of the resulting xml document?
rgasch
Forum Newbie
Posts: 3 Joined: Wed Sep 03, 2008 8:09 am
Post
by rgasch » Wed Sep 03, 2008 9:47 am
The file is simply truncated in the middle of a record. Here are the last 2 records (with the last one being truncated):
Code: Select all
<Product>
<id>1854</id>
<category_id>0</category_id>
<category2_id>0</category2_id>
<category3_id>0</category3_id>
<category4_id>0</category4_id>
<category5_id>0</category5_id>
<sku_set_id>0</sku_set_id>
<tax_id>0</tax_id>
<manufacturer_id>1</manufacturer_id>
<publisher_id>0</publisher_id>
<availability_id>0</availability_id>
<ship_info_id>0</ship_info_id>
<code>P 1226 5700</code>
<model></model>
<name>SOLE 1226 POLICE</name>
<author></author>
<isbn></isbn>
<url></url>
<image_orig></image_orig>
<image_large></image_large>
<image_med></image_med>
<image_small></image_small>
<image_extra1></image_extra1>
<image_extra2></image_extra2>
<image_extra3></image_extra3>
<image_extra4></image_extra4>
<image_extra5></image_extra5>
<is_featured>0</is_featured>
<is_downloadable>0</is_downloadable>
<download_file></download_file>
<description/>
<description_long/>
<features/>
<size></size>
<weight>0</weight>
<notes/>
<sort_order>0</sort_order>
<release_date>0000-00-00 00:00:00</release_date>
<price>195</price>
<vp_amount_1>0</vp_amount_1>
<vp_price_1>0</vp_price_1>
<vp_amount_2>0</vp_amount_2>
<vp_price_2>0</vp_price_2>
<vp_amount_3>0</vp_amount_3>
<vp_price_3>0</vp_price_3>
<discount>0</discount>
<inventory>0</inventory>
<oostock_override>0</oostock_override>
<coupon_code></coupon_code>
<sales_rank>0</sales_rank>
<sales_rank_category>0</sales_rank_category>
<view_count>0</view_count>
<status_cd>A</status_cd>
<obj_status>A</obj_status>
<cr_date>2008-09-03 13:17:32</cr_date>
<cr_uid>2</cr_uid>
<lu_date>2008-09-03 13:17:32</lu_date>
<lu_uid>2</lu_uid>
</Product>
<Product>
<id>1855</id>
<category_id>0</category_id>
<category2_id>0</category2_id>
<category3_id>0</category3_id>
<category4_id>0</category4_id>
<category5_id>0</category5_id>
<sku_set_id>0</sku_set_id>
<tax_id>0</tax_id>
<manufacturer_id>0</manufacturer_id>
<publisher_id>0</publisher_id>
<availability_id>0</availability_id>
<ship_info_id>0</ship_info_id>
<code>SGR 677 5616</code>
<model></model>
<name>SOLE 677 ROLLING</name>
<author></author>
<isbn></isbn>
<url></url>
<image_orig></image_orig>
<image_large></image_large>
<image_med></image_med>
<image_small></image_small>
<image_extra1></image_extra1>
<image_extra2></image_extra2>
<image_extra3></image_extra3>
<image_extra4></image_extra4>
<image_extra5></image_extra5>
<is_featured>0</is_featured>
<is_downloadable>0</
Greetings/Thanks
R
andyhoneycutt
Forum Contributor
Posts: 468 Joined: Wed Aug 27, 2008 10:02 am
Location: Idaho Falls
Post
by andyhoneycutt » Wed Sep 03, 2008 11:49 am
Ok, two questions: 1. How were you flushing the data from XmlWriter? 2. Does your selectFromDB function return an iterative result set that pops elements each time you access it? I'm wondering if you're running into a massively looping result-set.
-Andy
rgasch
Forum Newbie
Posts: 3 Joined: Wed Sep 03, 2008 8:09 am
Post
by rgasch » Wed Sep 03, 2008 3:04 pm
Hi,
1) I flush by doing a $writer->flush(); at the end of the XML generation. The code for this is as follows:
Code: Select all
.... xml generation code goes here ...
if ($this->_gzip) {
$xml = $writer->outputMemory();
$fp = gzopen ($this->_filename.'.gz', 'wb');
gzwrite ($fp, $xml);
gzclose ($fp);
} else {
$writer->flush();
}
The interesting thing here is that when I write to a gzipped file, the file also uncompresses to 3138986 bytes as well.
2) Yes, the call
Code: Select all
selectFromDB ('$page*$this->_pagesize, $this->_pagesize);
has paging built it. The 2 parameters specify the record offset to fetch from and the pagesize. The logic of paging through the DB records is correct and works (and even if it weren't, it wouldn't explain why the XML output is truncated in mid record).
I'm really stumped on this one, thanks for your help so far.
Greetings
R
andyhoneycutt
Forum Contributor
Posts: 468 Joined: Wed Aug 27, 2008 10:02 am
Location: Idaho Falls
Post
by andyhoneycutt » Wed Sep 03, 2008 4:26 pm
if you output the contents of $xml, is that cutoff as well? Silly question, I know, but I'm trying to pinpoint the issue without having the code at my fingertips