Page 1 of 1

using fsockopen and fgets, don't receive all data

Posted: Mon Feb 22, 2010 1:59 pm
by wutanghax0r
I've create a simple REST setup to have a script on one server request data from another.
If I access the REST server script manually it outputs all of the (json encoded) data without any problems.

my rest client script uses fsockopen to open the connection and then fgets to read the data in a loop.
It works perfectly with smaller amounts of data.
but after a certain point, It will add a hex value to the beginning of my json encoded data string and not get all of the data.

the data is all text json-encoded as an array of objects. With my current data, if I request 45 objects, It works successfully,
but at >= 46 it doesn't.
the actual size of the data is:
45 objects - 7,969 bytes
46 objects - 8,118 bytes

this data is all on one line, so my script loops through the response, reading the response header line by line with fgets()
and then reads the entire data object in one fgets() call. Is this a bad idea? Am I hitting some maximum size that can be read with one fgets() call?

Here's my code:

Code: Select all

 
        $sock = fsockopen($host, $port);
 
        fputs($sock, "GET $path HTTP/1.1\r\n");
        fputs($sock, "Host: $host\r\n");
        fputs($sock, "Content-type: application/x-www-form-urlencoded\r\n");
        fputs($sock, "Connection: close\r\n\r\n");
 
        // get the result.
        $result = "";
        while (!feof($sock)) 
        {   //length parameter increased to this crazy number while debugging :-/
            $result .= fgets($sock,9999768);
        }
 
        fclose($sock);
        return $result;
 
 

Re: using fsockopen and fgets, don't receive all data

Posted: Mon Feb 22, 2010 6:49 pm
by wutanghax0r
I was mistaken in my first post. It appears that my client script does receive all of the data, but before the line that has the bulk of the data on it, it adds the hex value of the size of that data.

for example, this is what I receive:

Code: Select all

HTTP/1.1 200 OK
Date: Tue, 23 Feb 2010 00:03:04 GMT
Server: Apache/2.2.6 (Fedora)
X-Powered-By: PHP/5.3.1
Connection: close
Transfer-Encoding: chunked
Content-Type: application/json
 
167cb
{*167CB bytes of json encoded data (including the curly brackets)*}
0
Note: I replaced the jason encoded data with the description between *'s

That hex value and the 0 at the end are not included when the data is less than about 8000bytes (decimal)

I seem to get the same results no matter how I set size (second parameter) using fgets() or fread()
I could add a conditional workaround to ignore the extra hex value when it's present and just read the json data, but I'd like to know why it is being included and if there is a way to get just the data without the extra values.

I also tested this with fgets() & fread(), just using fopen() and reading the same long json string from a hardcoded data file and it didn't add the extra hex value. It just read in the data as expected.

Re: using fsockopen and fgets, don't receive all data

Posted: Mon Feb 22, 2010 8:03 pm
by Benjamin
I would opt for a more simple solution.

You can use wget via a system call:

Code: Select all

$foo = system('wget http://www.myserver.com/file.txt ~',$output);
You can also use file_get_contents();

Code: Select all

$file = file_get_contents('http://example.com/file.ext');

Re: using fsockopen and fgets, don't receive all data

Posted: Tue Feb 23, 2010 6:02 pm
by wutanghax0r
Thanks for the response. If I run into more problems I'll switch to wget.

I haven't decided exactly how to deal with the data I'm receiving, but I figure out that the extra hex value is because the response is coming "Transfer-Encoding: chunked"
This means that it sends a size of the chunk followed by the data, followed by another chunk size followed by the data etc.
After the last chunk a chunk size of 0 is sent.

This is why I was receiving a hex size followed by my data, followed by a 0

I don't know how chunk sizes are determined but my data always comes in one chunk.

I wonder if there is a way to have my server side php script send back the data in an unchunked format or if there is a simple way to unchunk the data once I receive it. :)
Currently my sever side script just does this to send the data (the statusMessage() function gets the text status message that follows the http status code):

Code: Select all

    
$statusCode = 200;      
$status = 'HTTP/1.1 ' . $statusCode . ' ' . statusMessage($statusCode);
header($status);
header('Content-type: application/json');
// output body ($body holds my json encoded data string
echo $body;
 

Re: using fsockopen and fgets, don't receive all data

Posted: Tue Feb 23, 2010 6:39 pm
by wutanghax0r
I solved it myself. Not that anyone cares haha... but in case someone else happens to be running into the same problem...

I just changed the response header on the server script from:
# $status = 'HTTP/1.1 ' . $statusCode . ' ' . statusMessage($statusCode);
# header($status);

to:
# $status = 'HTTP/1.0 ' . $statusCode . ' ' . statusMessage($statusCode);
# header($status);

http 1.0 doesn't use the chunked transfer encoding.

Re: using fsockopen and fgets, don't receive all data

Posted: Tue Feb 23, 2010 7:19 pm
by requinix
Reading chunked data is really easy, and I'd suggest you do that rather than downgrade the protocol.
Wikipedia

First you get some hex data, maybe some whitespace, then a \r\n. Then comes the data. Then another \r\n and the sequence repeats. The end comes with a chunk size of zero, maybe some whitespace, and a final \r\n.

Code: Select all

function read_chunked($stream) {
    $data = "";
    do {
        for ($size = ""; $char != "\n"; $size .= $char) {
            if (feof($stream)) return $data;
            $char = fgetc($stream); // character in chunk size
        }
        if (feof($stream)) return $data;
 
        // size = [hex data]\r\n
        $size = base_convert(trim($size), 16, 10);
        if ($size == 0) break;
 
        $data .= fread($stream, $size);
        fread($stream, 2); // terminating \r\n
    } while (!feof($stream));
 
    return $data;
}
Untested.