Page 1 of 1

XMLWriter output file truncated?

Posted: Wed Sep 03, 2008 9:07 am
by rgasch
Hi,

I'm having a strange problem that I can't figure out. I'm trying to generate a large XML file using XMLWriter but the file I generate never grows beyond 3138986 bytes (whether I use the XMLWriter to write to a file/URI or generate the XML in memory and then pass it though gzwrite() doesn't affect the outcome; the (gunzipped) file never grows beyond 3138986 bytes.

Here's some code excepts of what I'm doing:

Code: Select all

$writer = new XMLWriter();
            if ($this->_gzip) {
                $writer->openMemory();
                $fp  = gzopen ($this->_filename.'.gz', 'wb');
            } else {
                $writer->openURI($this->_filename);
            }
 
        $writer->startDocument('1.0', 'ISO-8859-15');
        $writer->setIndent(4);
        $writer->startElement('Products');
 
        $count    = 0;
        $page     = 0;
        $objArray = new PNProductArray ();
        $colArray = getColumnsArray('table');
        do {
            $data = selectFromDB ('$page*$this->_pagesize, $this->_pagesize);
            foreach ($data as $dat) {
                $writer->startElement('Product');
                foreach ($colArray as $field) {
                    $writer->writeElement($field, $dat[$field]);
                }
                $writer->endElement();
                $count++;
            }
            $page++;
        } while ($data);
 
        $writer->endElement();
        $writer->endDocument();
 
        if ($this->_gzip) {
            $xml = $writer->outputMemory();
            $fp  = gzopen ($this->_filename.'.gz', 'wb');
            gzwrite ($fp, $xml);
            gzclose ($fp);
        } else {
            $writer->flush();
        }
 
In summary, no matter what I do, I can't seem to generate an XML file greater than 3138986 bytes. I've tried flushing the buffer after every DB fetch (if writing to a non-compressed file) but nothing helped. I'm not running into any (visible) PHP memory or processing time limits.

Any ideas would be much appreciated.

Thanks
Rob

Re: XMLWriter output file truncated?

Posted: Wed Sep 03, 2008 9:42 am
by andyhoneycutt
can you post a tail of the resulting xml document?

Re: XMLWriter output file truncated?

Posted: Wed Sep 03, 2008 9:47 am
by rgasch
The file is simply truncated in the middle of a record. Here are the last 2 records (with the last one being truncated):

Code: Select all

<Product>
  <id>1854</id>
  <category_id>0</category_id>
  <category2_id>0</category2_id>
  <category3_id>0</category3_id>
  <category4_id>0</category4_id>
  <category5_id>0</category5_id>
  <sku_set_id>0</sku_set_id>
  <tax_id>0</tax_id>
  <manufacturer_id>1</manufacturer_id>
  <publisher_id>0</publisher_id>
  <availability_id>0</availability_id>
  <ship_info_id>0</ship_info_id>
  <code>P 1226 5700</code>
  <model></model>
  <name>SOLE 1226 POLICE</name>
  <author></author>
  <isbn></isbn>
  <url></url>
  <image_orig></image_orig>
  <image_large></image_large>
  <image_med></image_med>
  <image_small></image_small>
  <image_extra1></image_extra1>
  <image_extra2></image_extra2>
  <image_extra3></image_extra3>
  <image_extra4></image_extra4>
  <image_extra5></image_extra5>
  <is_featured>0</is_featured>
  <is_downloadable>0</is_downloadable>
  <download_file></download_file>
  <description/>
  <description_long/>
  <features/>
  <size></size>
  <weight>0</weight>
  <notes/>
  <sort_order>0</sort_order>
  <release_date>0000-00-00 00:00:00</release_date>
  <price>195</price>
  <vp_amount_1>0</vp_amount_1>
  <vp_price_1>0</vp_price_1>
  <vp_amount_2>0</vp_amount_2>
  <vp_price_2>0</vp_price_2>
  <vp_amount_3>0</vp_amount_3>
  <vp_price_3>0</vp_price_3>
  <discount>0</discount>
  <inventory>0</inventory>
  <oostock_override>0</oostock_override>
  <coupon_code></coupon_code>
  <sales_rank>0</sales_rank>
  <sales_rank_category>0</sales_rank_category>
  <view_count>0</view_count>
  <status_cd>A</status_cd>
  <obj_status>A</obj_status>
  <cr_date>2008-09-03 13:17:32</cr_date>
  <cr_uid>2</cr_uid>
  <lu_date>2008-09-03 13:17:32</lu_date>
  <lu_uid>2</lu_uid>
 </Product>
 <Product>
  <id>1855</id>
  <category_id>0</category_id>
  <category2_id>0</category2_id>
  <category3_id>0</category3_id>
  <category4_id>0</category4_id>
  <category5_id>0</category5_id>
  <sku_set_id>0</sku_set_id>
  <tax_id>0</tax_id>
  <manufacturer_id>0</manufacturer_id>
  <publisher_id>0</publisher_id>
  <availability_id>0</availability_id>
  <ship_info_id>0</ship_info_id>
  <code>SGR 677 5616</code>
  <model></model>
  <name>SOLE 677 ROLLING</name>
  <author></author>
  <isbn></isbn>
  <url></url>
  <image_orig></image_orig>
  <image_large></image_large>
  <image_med></image_med>
  <image_small></image_small>
  <image_extra1></image_extra1>
  <image_extra2></image_extra2>
  <image_extra3></image_extra3>
  <image_extra4></image_extra4>
  <image_extra5></image_extra5>
  <is_featured>0</is_featured>
  <is_downloadable>0</
Greetings/Thanks
R

Re: XMLWriter output file truncated?

Posted: Wed Sep 03, 2008 11:49 am
by andyhoneycutt
Ok, two questions: 1. How were you flushing the data from XmlWriter? 2. Does your selectFromDB function return an iterative result set that pops elements each time you access it? I'm wondering if you're running into a massively looping result-set.

-Andy

Re: XMLWriter output file truncated?

Posted: Wed Sep 03, 2008 3:04 pm
by rgasch
Hi,

1) I flush by doing a $writer->flush(); at the end of the XML generation. The code for this is as follows:

Code: Select all

.... xml generation code goes here ...
if ($this->_gzip) {
             $xml = $writer->outputMemory();
             $fp  = gzopen ($this->_filename.'.gz', 'wb');
             gzwrite ($fp, $xml);
             gzclose ($fp);
         } else {
             $writer->flush();
         }
The interesting thing here is that when I write to a gzipped file, the file also uncompresses to 3138986 bytes as well.

2) Yes, the call

Code: Select all

selectFromDB ('$page*$this->_pagesize, $this->_pagesize);
has paging built it. The 2 parameters specify the record offset to fetch from and the pagesize. The logic of paging through the DB records is correct and works (and even if it weren't, it wouldn't explain why the XML output is truncated in mid record).

I'm really stumped on this one, thanks for your help so far.

Greetings
R

Re: XMLWriter output file truncated?

Posted: Wed Sep 03, 2008 4:26 pm
by andyhoneycutt
if you output the contents of $xml, is that cutoff as well? Silly question, I know, but I'm trying to pinpoint the issue without having the code at my fingertips :)