last_json_error(), problem with JSON from Google

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
User avatar
zeus1
Forum Newbie
Posts: 22
Joined: Tue May 18, 2010 10:45 pm
Location: /Under/The/AppleTree.png

last_json_error(), problem with JSON from Google

Post by zeus1 »

I am having a problem decoding JSON from google. I get the JSON using libcurl and try to parse it, but I keep getting malformed syntax error. I do not believe I am getting the error because of recursion depth. To test what I just stated I only removed all JSON results except for the first. Can someone check to see if they can decode the JSON data? The following is my script:

Code: Select all

<?
$ch = curl_init(REDACTED);

$today  = date('d_M_Y_H:i:s');

$filename = "financialJson_" . $today . ".json";

$fp = fopen($filename, "w");

curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, "Accept: application/json");

curl_exec($ch);
curl_close($ch);
fclose($fp);

//read the written file that contain json data
$currentFile = fread(fopen($filename, 'r'),filesize($filename) );

//decode the json data, second param is true to make it an assoc array

$json = json_decode($currentFile,true);

//print out the array
//echo $currentFile;
var_dump($json);
echo "File name: " . $filename;
echo $currentFile;
//deugging
 switch(json_last_error())
    {
        case JSON_ERROR_DEPTH:
            echo ' - Maximum stack depth exceeded';
        break;
        case JSON_ERROR_CTRL_CHAR:
            echo ' - Unexpected control character found';
        break;
        case JSON_ERROR_SYNTAX:
            echo ' - Syntax error, malformed JSON';
        break;
        case JSON_ERROR_NONE:
            echo ' - No errors';
        break;
    }


?>
Best Regards,
ZEUS
Last edited by zeus1 on Wed May 19, 2010 2:13 pm, edited 1 time in total.
User avatar
Benjamin
Site Administrator
Posts: 6935
Joined: Sun May 19, 2002 10:24 pm

Re: last_json_error(), problem with JSON from Google

Post by Benjamin »

The variable $currentFile does not contain json code, because it was never written to the file. curl_exec is sending output to the browser, because you are not capturing it.

http://us3.php.net/manual/en/function.curl-exec.php
User avatar
zeus1
Forum Newbie
Posts: 22
Joined: Tue May 18, 2010 10:45 pm
Location: /Under/The/AppleTree.png

Re: last_json_error(), problem with JSON from Google

Post by zeus1 »

Did you try running the script? This option

Code: Select all

curl_setopt($ch, CURLOPT_FILE, $fp);
writes the output to a file. At the end of the script I echo $currentFile and it prints out the json file. It might be confusing because I close the file and then reopen it for decoding. I think it might be google's syntax of json. I attached a sample of the output json that I receive.
{
"start" : "0",
"num" : "90",
"num_company_results" : "99",
"num_mf_results" : "1",
"num_all_results" : "",
"original_query" : "((exchange:NYSE) OR (exchange:NASDAQ) OR (exchange:AMEX)) [(MarketCap \x3E 4250000000 | MarketCap = 4250000000) \x26 (MarketCap \x3C 2270000000000 | MarketCap = 2270000000000) \x26 (PE \x3E 0 | PE = 0) \x26 (PE \x3C 34.18 | PE = 34.18) \x26 (Price13WeekPercChange \x3E -16.39 | Price13WeekPercChange = -16.39) \x26 (Price13WeekPercChange \x3C 0.25 | Price13WeekPercChange = 0.25) \x26 (QuoteLast \x3E 0 | QuoteLast = 0) \x26 (QuoteLast \x3C 115326 | QuoteLast = 115326) \x26 (QuotePercChange \x3E -66.75 | QuotePercChange = -66.75) \x26 (QuotePercChange \x3C 60.81 | QuotePercChange = 60.81) \x26 (AverageVolume \x3E 1090000 | AverageVolume = 1090000) \x26 (AverageVolume \x3C 802450000 | AverageVolume = 802450000)]",
"query_for_display" : "((exchange:NYSE) OR (exchange:NASDAQ) OR (exchange:AMEX)) [(MarketCap > 4250000000 | MarketCap = 4250000000) & (MarketCap < 2270000000000 | MarketCap = 2270000000000) & (PE > 0 | PE = 0) & (PE < 34.18 | PE = 34.18) & (Price13WeekPercChange > -16.39 | Price13WeekPercChange = -16.39) & (Price13WeekPercChange < 0.25 | Price13WeekPercChange = 0.25) & (QuoteLast > 0 | QuoteLast = 0) & (QuoteLast < 115326 | QuoteLast = 115326) & (QuotePercChange > -66.75 | QuotePercChange = -66.75) & (QuotePercChange < 60.81 | QuotePercChange = 60.81) & (AverageVolume > 1090000 | AverageVolume = 1090000) & (AverageVolume < 802450000 | AverageVolume = 802450000)]",
"results_type" : "COMPANY",
"searchresults" :
[
{
"title" : "Abbott Laboratories",
"id" : "145",
"is_active" : "",
"ticker" : "ABT",
"exchange" : "NYSE",
"is_supported_exchange" : "",
"columns" : [
{
"display_name": "",
"value" : "74.75B",
"field" : "MarketCap",
"sort_order" : ""
}
, {
"display_name": "",
"value" : "14.19",
"field" : "PE",
"sort_order" : ""
}
, {
"display_name": "",
"value" : "-9.21",
"field" : "Price13WeekPercChange",
"sort_order" : ""
}
, {
"display_name": "",
"value" : "48.43",
"field" : "QuoteLast",
"sort_order" : ""
}
, {
"display_name": "",
"value" : "-0.14",
"field" : "QuotePercChange",
"sort_order" : ""
}
, {
"display_name": "",
"value" : "8.96M",
"field" : "AverageVolume",
"sort_order" : ""
}
]
}
, {
"title" : "Agrium Inc. (USA)",
"id" : "661784",
"is_active" : "",
"ticker" : "AGU",
"exchange" : "NYSE",
"is_supported_exchange" : "",
"columns" : [
{
"display_name": "",
"value" : "8.76B",
"field" : "MarketCap",
"sort_order" : ""
}
, {
"display_name": "",
"value" : "20.92",
"field" : "PE",
"sort_order" : ""
}
, {
"display_name": "",
"value" : "-10.99",
"field" : "Price13WeekPercChange",
"sort_order" : ""
}
, {
"display_name": "",
"value" : "55.82",
"field" : "QuoteLast",
"sort_order" : ""
}
, {
"display_name": "",
"value" : "-1.05",
"field" : "QuotePercChange",
"sort_order" : ""
}
, {
"display_name": "",
"value" : "1.91M",
"field" : "AverageVolume",
"sort_order" : ""
}
]
}
]
User avatar
zeus1
Forum Newbie
Posts: 22
Joined: Tue May 18, 2010 10:45 pm
Location: /Under/The/AppleTree.png

Re: last_json_error(), problem with JSON from Google

Post by zeus1 »

I got it working. I reviewed the JSON and figured out that the following lines were preventing the json from being decoded
{
"start" : "0",
"num" : "90",
"num_company_results" : "99",
"num_mf_results" : "1",
"num_all_results" : "",
"original_query" : "((exchange:NYSE) OR (exchange:NASDAQ) OR (exchange:AMEX)) [(MarketCap \x3E 4250000000 | MarketCap = 4250000000) \x26 (MarketCap \x3C 2270000000000 | MarketCap = 2270000000000) \x26 (PE \x3E 0 | PE = 0) \x26 (PE \x3C 34.18 | PE = 34.18) \x26 (Price13WeekPercChange \x3E -16.39 | Price13WeekPercChange = -16.39) \x26 (Price13WeekPercChange \x3C 0.25 | Price13WeekPercChange = 0.25) \x26 (QuoteLast \x3E 0 | QuoteLast = 0) \x26 (QuoteLast \x3C 115326 | QuoteLast = 115326) \x26 (QuotePercChange \x3E -66.75 | QuotePercChange = -66.75) \x26 (QuotePercChange \x3C 60.81 | QuotePercChange = 60.81) \x26 (AverageVolume \x3E 1090000 | AverageVolume = 1090000) \x26 (AverageVolume \x3C 802450000 | AverageVolume = 802450000)]",
"query_for_display" : "((exchange:NYSE) OR (exchange:NASDAQ) OR (exchange:AMEX)) [(MarketCap > 4250000000 | MarketCap = 4250000000) & (MarketCap < 2270000000000 | MarketCap = 2270000000000) & (PE > 0 | PE = 0) & (PE < 34.18 | PE = 34.18) & (Price13WeekPercChange > -16.39 | Price13WeekPercChange = -16.39) & (Price13WeekPercChange < 0.25 | Price13WeekPercChange = 0.25) & (QuoteLast > 0 | QuoteLast = 0) & (QuoteLast < 115326 | QuoteLast = 115326) & (QuotePercChange > -66.75 | QuotePercChange = -66.75) & (QuotePercChange < 60.81 | QuotePercChange = 60.81) & (AverageVolume > 1090000 | AverageVolume = 1090000) & (AverageVolume < 802450000 | AverageVolume = 802450000)]",
"results_type" : "COMPANY",
"searchresults" :
Can anyone suggest the best way to remove these lines consistently, or rather ensure I only get the working json code? Maybe if I start writing only after the second [ ?
User avatar
John Cartwright
Site Admin
Posts: 11470
Joined: Tue Dec 23, 2003 2:10 am
Location: Toronto
Contact:

Re: last_json_error(), problem with JSON from Google

Post by John Cartwright »

Just a thought.. try running the raw json string through addslashes() prior to decoding it.. and stripslashes() on values.
User avatar
Benjamin
Site Administrator
Posts: 6935
Joined: Sun May 19, 2002 10:24 pm

Re: last_json_error(), problem with JSON from Google

Post by Benjamin »

The JSON decoded fine for me, without issue.
User avatar
zeus1
Forum Newbie
Posts: 22
Joined: Tue May 18, 2010 10:45 pm
Location: /Under/The/AppleTree.png

Re: last_json_error(), problem with JSON from Google

Post by zeus1 »

I discovered the JSON file was reaching recursion depth. Does anyone have any hints as how I should go about to fix the problem. The json file can be downloaded from the following link. http://www.google.com/finance?start=0&n ... son&noIL=1 Can anyone decode without any trouble? Benjamin, did you use the following link to get the entire JSON file? What I had was abridged. Can you let me know if you can decode using the JSON file from the link above. If so, please let me know. I think it is a problem with my recursion limit. I tried messing around with it inline init_set(), to no avail. Then I tried setting pcre.recursion_limit and memory limit really high in the php.ini file.
User avatar
John Cartwright
Site Admin
Posts: 11470
Joined: Tue Dec 23, 2003 2:10 am
Location: Toronto
Contact:

Re: last_json_error(), problem with JSON from Google

Post by John Cartwright »

Running this test script:

Code: Select all

<?php

$ch = curl_init('http://www.google.com/finance?start=0&num=90&gl=us&hl=en&q=%28%28exchange%3ANYSE%29%20OR%20%28exchange%3ANASDAQ%29%20OR%20%28exchange%3AAMEX%29%29%20%5B%28MarketCap%20%3E%204250000000%20%7C%20MarketCap%20%3D%204250000000%29%20%26%20%28MarketCap%20%3C%202270000000000%20%7C%20MarketCap%20%3D%202270000000000%29%20%26%20%28PE%20%3E%200%20%7C%20PE%20%3D%200%29%20%26%20%28PE%20%3C%2034.18%20%7C%20PE%20%3D%2034.18%29%20%26%20%28Price13WeekPercChange%20%3E%20-16.39%20%7C%20Price13WeekPercChange%20%3D%20-16.39%29%20%26%20%28Price13WeekPercChange%20%3C%200.25%20%7C%20Price13WeekPercChange%20%3D%200.25%29%20%26%20%28QuoteLast%20%3E%200%20%7C%20QuoteLast%20%3D%200%29%20%26%20%28QuoteLast%20%3C%20115326%20%7C%20QuoteLast%20%3D%20115326%29%20%26%20%28QuotePercChange%20%3E%20-66.75%20%7C%20QuotePercChange%20%3D%20-66.75%29%20%26%20%28QuotePercChange%20%3C%2060.81%20%7C%20QuotePercChange%20%3D%2060.81%29%20%26%20%28AverageVolume%20%3E%201090000%20%7C%20AverageVolume%20%3D%201090000%29%20%26%20%28AverageVolume%20%3C%20802450000%20%7C%20AverageVolume%20%3D%20802450000%29%5D&restype=company&output=json&noIL=1');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$json = curl_exec($ch);

echo '<pre>';
echo 'Json result: '; var_dump(json_decode($json, true));
echo '</pre>';

switch(json_last_error()) {
    case JSON_ERROR_DEPTH:
        echo 'Maximum stack depth exceeded';
    break;
    case JSON_ERROR_CTRL_CHAR:
        echo 'Unexpected control character found';
    break;
    case JSON_ERROR_SYNTAX:
        echo 'Syntax error, malformed JSON';
    break;
    case JSON_ERROR_NONE:
        echo 'No errors';
    break;
}

?>
I got the following response

[text]Json result: NULL

Syntax error, malformed JSON[/text]

I would have posted the json string as well, however it is over double the size of our max allowed character count to post. But it appears there is something wrong with the syntax of the string, and not the depth.

How did you determine you were having a recursion issue?
User avatar
Weirdan
Moderator
Posts: 5978
Joined: Mon Nov 03, 2003 6:13 pm
Location: Odessa, Ukraine

Re: last_json_error(), problem with JSON from Google

Post by Weirdan »

The json is invalid indeed - they use \x escape code that is not allowed in json strings. This simple example would fail as well:

Code: Select all

{"original_query" : "\x3E"}
User avatar
zeus1
Forum Newbie
Posts: 22
Joined: Tue May 18, 2010 10:45 pm
Location: /Under/The/AppleTree.png

Re: last_json_error(), problem with JSON from Google

Post by zeus1 »

John Cartwright wrote:Running this test script:

Code: Select all

<?php

$ch = curl_init('http://www.google.com/finance?start=0&num=90&gl=us&hl=en&q=%28%28exchange%3ANYSE%29%20OR%20%28exchange%3ANASDAQ%29%20OR%20%28exchange%3AAMEX%29%29%20%5B%28MarketCap%20%3E%204250000000%20%7C%20MarketCap%20%3D%204250000000%29%20%26%20%28MarketCap%20%3C%202270000000000%20%7C%20MarketCap%20%3D%202270000000000%29%20%26%20%28PE%20%3E%200%20%7C%20PE%20%3D%200%29%20%26%20%28PE%20%3C%2034.18%20%7C%20PE%20%3D%2034.18%29%20%26%20%28Price13WeekPercChange%20%3E%20-16.39%20%7C%20Price13WeekPercChange%20%3D%20-16.39%29%20%26%20%28Price13WeekPercChange%20%3C%200.25%20%7C%20Price13WeekPercChange%20%3D%200.25%29%20%26%20%28QuoteLast%20%3E%200%20%7C%20QuoteLast%20%3D%200%29%20%26%20%28QuoteLast%20%3C%20115326%20%7C%20QuoteLast%20%3D%20115326%29%20%26%20%28QuotePercChange%20%3E%20-66.75%20%7C%20QuotePercChange%20%3D%20-66.75%29%20%26%20%28QuotePercChange%20%3C%2060.81%20%7C%20QuotePercChange%20%3D%2060.81%29%20%26%20%28AverageVolume%20%3E%201090000%20%7C%20AverageVolume%20%3D%201090000%29%20%26%20%28AverageVolume%20%3C%20802450000%20%7C%20AverageVolume%20%3D%20802450000%29%5D&restype=company&output=json&noIL=1');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$json = curl_exec($ch);

echo '<pre>';
echo 'Json result: '; var_dump(json_decode($json, true));
echo '</pre>';

switch(json_last_error()) {
    case JSON_ERROR_DEPTH:
        echo 'Maximum stack depth exceeded';
    break;
    case JSON_ERROR_CTRL_CHAR:
        echo 'Unexpected control character found';
    break;
    case JSON_ERROR_SYNTAX:
        echo 'Syntax error, malformed JSON';
    break;
    case JSON_ERROR_NONE:
        echo 'No errors';
    break;
}

?>
I got the following response

[text]Json result: NULL

Syntax error, malformed JSON[/text]

I would have posted the json string as well, however it is over double the size of our max allowed character count to post. But it appears there is something wrong with the syntax of the string, and not the depth.

How did you determine you were having a recursion issue?
How I determined it was a recursion issue is when I wrote the file I went in and deleted the lines
{
"start" : "0",
"num" : "90",
"num_company_results" : "99",
"num_mf_results" : "1",
"num_all_results" : "",
"original_query" : "((exchange:NYSE) OR (exchange:NASDAQ) OR (exchange:AMEX)) [(MarketCap \x3E 4250000000 | MarketCap = 4250000000) \x26 (MarketCap \x3C 2270000000000 | MarketCap = 2270000000000) \x26 (PE \x3E 0 | PE = 0) \x26 (PE \x3C 34.18 | PE = 34.18) \x26 (Price13WeekPercChange \x3E -16.39 | Price13WeekPercChange = -16.39) \x26 (Price13WeekPercChange \x3C 0.25 | Price13WeekPercChange = 0.25) \x26 (QuoteLast \x3E 0 | QuoteLast = 0) \x26 (QuoteLast \x3C 115326 | QuoteLast = 115326) \x26 (QuotePercChange \x3E -66.75 | QuotePercChange = -66.75) \x26 (QuotePercChange \x3C 60.81 | QuotePercChange = 60.81) \x26 (AverageVolume \x3E 1090000 | AverageVolume = 1090000) \x26 (AverageVolume \x3C 802450000 | AverageVolume = 802450000)]",
"query_for_display" : "((exchange:NYSE) OR (exchange:NASDAQ) OR (exchange:AMEX)) [(MarketCap > 4250000000 | MarketCap = 4250000000) & (MarketCap < 2270000000000 | MarketCap = 2270000000000) & (PE > 0 | PE = 0) & (PE < 34.18 | PE = 34.18) & (Price13WeekPercChange > -16.39 | Price13WeekPercChange = -16.39) & (Price13WeekPercChange < 0.25 | Price13WeekPercChange = 0.25) & (QuoteLast > 0 | QuoteLast = 0) & (QuoteLast < 115326 | QuoteLast = 115326) & (QuotePercChange > -66.75 | QuotePercChange = -66.75) & (QuotePercChange < 60.81 | QuotePercChange = 60.81) & (AverageVolume > 1090000 | AverageVolume = 1090000) & (AverageVolume < 802450000 | AverageVolume = 802450000)]",
"results_type" : "COMPANY",
"searchresults" :
and I deleted the last lines in the json file :
,
"mf_searchresults" :
[
]
}
And I only left the important json data. It still didn't work. Then, I removed all of the records, except for the first few. And it worked. After that I kept editing the size of the pertinent financial data. At certain sizes it works, and others no. Try removing those lines in the begging and end. It should be valid JSON. When benjamin said he decoded with no issue, I assumed everything was valid, and my problem was actually a memory one.
User avatar
Weirdan
Moderator
Posts: 5978
Joined: Mon Nov 03, 2003 6:13 pm
Location: Odessa, Ukraine

Re: last_json_error(), problem with JSON from Google

Post by Weirdan »

Works for me:

Code: Select all

<?php

$ch = curl_init('http://www.google.com/finance?start=0&num=90&gl=us&hl=en&q=%28%28exchange%3ANYSE%29%20OR%20%28exchange%3ANASDAQ%29%20OR%20%28exchange%3AAMEX%29%29%20%5B%28MarketCap%20%3E%204250000000%20%7C%20MarketCap%20%3D%204250000000%29%20%26%20%28MarketCap%20%3C%202270000000000%20%7C%20MarketCap%20%3D%202270000000000%29%20%26%20%28PE%20%3E%200%20%7C%20PE%20%3D%200%29%20%26%20%28PE%20%3C%2034.18%20%7C%20PE%20%3D%2034.18%29%20%26%20%28Price13WeekPercChange%20%3E%20-16.39%20%7C%20Price13WeekPercChange%20%3D%20-16.39%29%20%26%20%28Price13WeekPercChange%20%3C%200.25%20%7C%20Price13WeekPercChange%20%3D%200.25%29%20%26%20%28QuoteLast%20%3E%200%20%7C%20QuoteLast%20%3D%200%29%20%26%20%28QuoteLast%20%3C%20115326%20%7C%20QuoteLast%20%3D%20115326%29%20%26%20%28QuotePercChange%20%3E%20-66.75%20%7C%20QuotePercChange%20%3D%20-66.75%29%20%26%20%28QuotePercChange%20%3C%2060.81%20%7C%20QuotePercChange%20%3D%2060.81%29%20%26%20%28AverageVolume%20%3E%201090000%20%7C%20AverageVolume%20%3D%201090000%29%20%26%20%28AverageVolume%20%3C%20802450000%20%7C%20AverageVolume%20%3D%20802450000%29%5D&restype=company&output=json&noIL=1');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$json = curl_exec($ch);

echo '<pre>';
echo 'Json result: '; var_dump(json_decode(preg_replace('#\\\\x[0-9a-fA-F]{2,2}#', '', $json), true));
echo '</pre>';

switch(json_last_error()) {
    case JSON_ERROR_DEPTH:
        echo 'Maximum stack depth exceeded';
    break;
    case JSON_ERROR_CTRL_CHAR:
        echo 'Unexpected control character found';
    break;
    case JSON_ERROR_SYNTAX:
        echo 'Syntax error, malformed JSON';
    break;
    case JSON_ERROR_NONE:
        echo 'No errors';
    break;
}

User avatar
zeus1
Forum Newbie
Posts: 22
Joined: Tue May 18, 2010 10:45 pm
Location: /Under/The/AppleTree.png

Re: last_json_error(), problem with JSON from Google

Post by zeus1 »

Weirdan, thats great news for me! What kind of setting do you have on your php server, like memory and recursion limits? What do you think could be causing mine to return null? My php settings are default.
User avatar
zeus1
Forum Newbie
Posts: 22
Joined: Tue May 18, 2010 10:45 pm
Location: /Under/The/AppleTree.png

Re: last_json_error(), problem with JSON from Google

Post by zeus1 »

Never mind. You did it! You cleaning of the JSON using

Code: Select all

preg_replace('#\\\\x[0-9a-fA-F]{2,2}#', '', $json)
fixed it. My original script works just fine! Cheers Weirdan!
Post Reply