Page 1 of 1
last_json_error(), problem with JSON from Google
Posted: Tue May 18, 2010 10:56 pm
by zeus1
I am having a problem decoding JSON from google. I get the JSON using libcurl and try to parse it, but I keep getting malformed syntax error. I do not believe I am getting the error because of recursion depth. To test what I just stated I only removed all JSON results except for the first. Can someone check to see if they can decode the JSON data? The following is my script:
Code: Select all
<?
$ch = curl_init(REDACTED);
$today = date('d_M_Y_H:i:s');
$filename = "financialJson_" . $today . ".json";
$fp = fopen($filename, "w");
curl_setopt($ch, CURLOPT_FILE, $fp);
curl_setopt($ch, CURLOPT_HEADER, "Accept: application/json");
curl_exec($ch);
curl_close($ch);
fclose($fp);
//read the written file that contain json data
$currentFile = fread(fopen($filename, 'r'),filesize($filename) );
//decode the json data, second param is true to make it an assoc array
$json = json_decode($currentFile,true);
//print out the array
//echo $currentFile;
var_dump($json);
echo "File name: " . $filename;
echo $currentFile;
//deugging
switch(json_last_error())
{
case JSON_ERROR_DEPTH:
echo ' - Maximum stack depth exceeded';
break;
case JSON_ERROR_CTRL_CHAR:
echo ' - Unexpected control character found';
break;
case JSON_ERROR_SYNTAX:
echo ' - Syntax error, malformed JSON';
break;
case JSON_ERROR_NONE:
echo ' - No errors';
break;
}
?>
Best Regards,
ZEUS
Re: last_json_error(), problem with JSON from Google
Posted: Wed May 19, 2010 4:29 am
by Benjamin
The variable $currentFile does not contain json code, because it was never written to the file. curl_exec is sending output to the browser, because you are not capturing it.
http://us3.php.net/manual/en/function.curl-exec.php
Re: last_json_error(), problem with JSON from Google
Posted: Wed May 19, 2010 1:00 pm
by zeus1
Did you try running the script? This option
Code: Select all
curl_setopt($ch, CURLOPT_FILE, $fp);
writes the output to a file. At the end of the script I echo $currentFile and it prints out the json file. It might be confusing because I close the file and then reopen it for decoding. I think it might be google's syntax of json. I attached a sample of the output json that I receive.
{
"start" : "0",
"num" : "90",
"num_company_results" : "99",
"num_mf_results" : "1",
"num_all_results" : "",
"original_query" : "((exchange:NYSE) OR (exchange:NASDAQ) OR (exchange:AMEX)) [(MarketCap \x3E 4250000000 | MarketCap = 4250000000) \x26 (MarketCap \x3C 2270000000000 | MarketCap = 2270000000000) \x26 (PE \x3E 0 | PE = 0) \x26 (PE \x3C 34.18 | PE = 34.18) \x26 (Price13WeekPercChange \x3E -16.39 | Price13WeekPercChange = -16.39) \x26 (Price13WeekPercChange \x3C 0.25 | Price13WeekPercChange = 0.25) \x26 (QuoteLast \x3E 0 | QuoteLast = 0) \x26 (QuoteLast \x3C 115326 | QuoteLast = 115326) \x26 (QuotePercChange \x3E -66.75 | QuotePercChange = -66.75) \x26 (QuotePercChange \x3C 60.81 | QuotePercChange = 60.81) \x26 (AverageVolume \x3E 1090000 | AverageVolume = 1090000) \x26 (AverageVolume \x3C 802450000 | AverageVolume = 802450000)]",
"query_for_display" : "((exchange:NYSE) OR (exchange:NASDAQ) OR (exchange:AMEX)) [(MarketCap > 4250000000 | MarketCap = 4250000000) & (MarketCap < 2270000000000 | MarketCap = 2270000000000) & (PE > 0 | PE = 0) & (PE < 34.18 | PE = 34.18) & (Price13WeekPercChange > -16.39 | Price13WeekPercChange = -16.39) & (Price13WeekPercChange < 0.25 | Price13WeekPercChange = 0.25) & (QuoteLast > 0 | QuoteLast = 0) & (QuoteLast < 115326 | QuoteLast = 115326) & (QuotePercChange > -66.75 | QuotePercChange = -66.75) & (QuotePercChange < 60.81 | QuotePercChange = 60.81) & (AverageVolume > 1090000 | AverageVolume = 1090000) & (AverageVolume < 802450000 | AverageVolume = 802450000)]",
"results_type" : "COMPANY",
"searchresults" :
[
{
"title" : "Abbott Laboratories",
"id" : "145",
"is_active" : "",
"ticker" : "ABT",
"exchange" : "NYSE",
"is_supported_exchange" : "",
"columns" : [
{
"display_name": "",
"value" : "74.75B",
"field" : "MarketCap",
"sort_order" : ""
}
, {
"display_name": "",
"value" : "14.19",
"field" : "PE",
"sort_order" : ""
}
, {
"display_name": "",
"value" : "-9.21",
"field" : "Price13WeekPercChange",
"sort_order" : ""
}
, {
"display_name": "",
"value" : "48.43",
"field" : "QuoteLast",
"sort_order" : ""
}
, {
"display_name": "",
"value" : "-0.14",
"field" : "QuotePercChange",
"sort_order" : ""
}
, {
"display_name": "",
"value" : "8.96M",
"field" : "AverageVolume",
"sort_order" : ""
}
]
}
, {
"title" : "Agrium Inc. (USA)",
"id" : "661784",
"is_active" : "",
"ticker" : "AGU",
"exchange" : "NYSE",
"is_supported_exchange" : "",
"columns" : [
{
"display_name": "",
"value" : "8.76B",
"field" : "MarketCap",
"sort_order" : ""
}
, {
"display_name": "",
"value" : "20.92",
"field" : "PE",
"sort_order" : ""
}
, {
"display_name": "",
"value" : "-10.99",
"field" : "Price13WeekPercChange",
"sort_order" : ""
}
, {
"display_name": "",
"value" : "55.82",
"field" : "QuoteLast",
"sort_order" : ""
}
, {
"display_name": "",
"value" : "-1.05",
"field" : "QuotePercChange",
"sort_order" : ""
}
, {
"display_name": "",
"value" : "1.91M",
"field" : "AverageVolume",
"sort_order" : ""
}
]
}
]
Re: last_json_error(), problem with JSON from Google
Posted: Wed May 19, 2010 2:05 pm
by zeus1
I got it working. I reviewed the JSON and figured out that the following lines were preventing the json from being decoded
{
"start" : "0",
"num" : "90",
"num_company_results" : "99",
"num_mf_results" : "1",
"num_all_results" : "",
"original_query" : "((exchange:NYSE) OR (exchange:NASDAQ) OR (exchange:AMEX)) [(MarketCap \x3E 4250000000 | MarketCap = 4250000000) \x26 (MarketCap \x3C 2270000000000 | MarketCap = 2270000000000) \x26 (PE \x3E 0 | PE = 0) \x26 (PE \x3C 34.18 | PE = 34.18) \x26 (Price13WeekPercChange \x3E -16.39 | Price13WeekPercChange = -16.39) \x26 (Price13WeekPercChange \x3C 0.25 | Price13WeekPercChange = 0.25) \x26 (QuoteLast \x3E 0 | QuoteLast = 0) \x26 (QuoteLast \x3C 115326 | QuoteLast = 115326) \x26 (QuotePercChange \x3E -66.75 | QuotePercChange = -66.75) \x26 (QuotePercChange \x3C 60.81 | QuotePercChange = 60.81) \x26 (AverageVolume \x3E 1090000 | AverageVolume = 1090000) \x26 (AverageVolume \x3C 802450000 | AverageVolume = 802450000)]",
"query_for_display" : "((exchange:NYSE) OR (exchange:NASDAQ) OR (exchange:AMEX)) [(MarketCap > 4250000000 | MarketCap = 4250000000) & (MarketCap < 2270000000000 | MarketCap = 2270000000000) & (PE > 0 | PE = 0) & (PE < 34.18 | PE = 34.18) & (Price13WeekPercChange > -16.39 | Price13WeekPercChange = -16.39) & (Price13WeekPercChange < 0.25 | Price13WeekPercChange = 0.25) & (QuoteLast > 0 | QuoteLast = 0) & (QuoteLast < 115326 | QuoteLast = 115326) & (QuotePercChange > -66.75 | QuotePercChange = -66.75) & (QuotePercChange < 60.81 | QuotePercChange = 60.81) & (AverageVolume > 1090000 | AverageVolume = 1090000) & (AverageVolume < 802450000 | AverageVolume = 802450000)]",
"results_type" : "COMPANY",
"searchresults" :
Can anyone suggest the best way to remove these lines consistently, or rather ensure I only get the working json code? Maybe if I start writing only after the second [ ?
Re: last_json_error(), problem with JSON from Google
Posted: Wed May 19, 2010 2:39 pm
by John Cartwright
Just a thought.. try running the raw json string through addslashes() prior to decoding it.. and stripslashes() on values.
Re: last_json_error(), problem with JSON from Google
Posted: Wed May 19, 2010 4:19 pm
by Benjamin
The JSON decoded fine for me, without issue.
Re: last_json_error(), problem with JSON from Google
Posted: Thu May 20, 2010 1:28 am
by zeus1
I discovered the JSON file was reaching recursion depth. Does anyone have any hints as how I should go about to fix the problem. The json file can be downloaded from the following link.
http://www.google.com/finance?start=0&n ... son&noIL=1 Can anyone decode without any trouble? Benjamin, did you use the following link to get the entire JSON file? What I had was abridged. Can you let me know if you can decode using the JSON file from the link above. If so, please let me know. I think it is a problem with my recursion limit. I tried messing around with it inline init_set(), to no avail. Then I tried setting pcre.recursion_limit and memory limit really high in the php.ini file.
Re: last_json_error(), problem with JSON from Google
Posted: Thu May 20, 2010 1:47 am
by John Cartwright
Running this test script:
Code: Select all
<?php
$ch = curl_init('http://www.google.com/finance?start=0&num=90&gl=us&hl=en&q=%28%28exchange%3ANYSE%29%20OR%20%28exchange%3ANASDAQ%29%20OR%20%28exchange%3AAMEX%29%29%20%5B%28MarketCap%20%3E%204250000000%20%7C%20MarketCap%20%3D%204250000000%29%20%26%20%28MarketCap%20%3C%202270000000000%20%7C%20MarketCap%20%3D%202270000000000%29%20%26%20%28PE%20%3E%200%20%7C%20PE%20%3D%200%29%20%26%20%28PE%20%3C%2034.18%20%7C%20PE%20%3D%2034.18%29%20%26%20%28Price13WeekPercChange%20%3E%20-16.39%20%7C%20Price13WeekPercChange%20%3D%20-16.39%29%20%26%20%28Price13WeekPercChange%20%3C%200.25%20%7C%20Price13WeekPercChange%20%3D%200.25%29%20%26%20%28QuoteLast%20%3E%200%20%7C%20QuoteLast%20%3D%200%29%20%26%20%28QuoteLast%20%3C%20115326%20%7C%20QuoteLast%20%3D%20115326%29%20%26%20%28QuotePercChange%20%3E%20-66.75%20%7C%20QuotePercChange%20%3D%20-66.75%29%20%26%20%28QuotePercChange%20%3C%2060.81%20%7C%20QuotePercChange%20%3D%2060.81%29%20%26%20%28AverageVolume%20%3E%201090000%20%7C%20AverageVolume%20%3D%201090000%29%20%26%20%28AverageVolume%20%3C%20802450000%20%7C%20AverageVolume%20%3D%20802450000%29%5D&restype=company&output=json&noIL=1');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$json = curl_exec($ch);
echo '<pre>';
echo 'Json result: '; var_dump(json_decode($json, true));
echo '</pre>';
switch(json_last_error()) {
case JSON_ERROR_DEPTH:
echo 'Maximum stack depth exceeded';
break;
case JSON_ERROR_CTRL_CHAR:
echo 'Unexpected control character found';
break;
case JSON_ERROR_SYNTAX:
echo 'Syntax error, malformed JSON';
break;
case JSON_ERROR_NONE:
echo 'No errors';
break;
}
?>
I got the following response
[text]Json result: NULL
Syntax error, malformed JSON[/text]
I would have posted the json string as well, however it is over double the size of our max allowed character count to post. But it appears there is something wrong with the syntax of the string, and not the depth.
How did you determine you were having a recursion issue?
Re: last_json_error(), problem with JSON from Google
Posted: Thu May 20, 2010 3:35 am
by Weirdan
The json is invalid indeed - they use \x escape code that is not allowed in json strings. This simple example would fail as well:
Re: last_json_error(), problem with JSON from Google
Posted: Thu May 20, 2010 1:57 pm
by zeus1
John Cartwright wrote:Running this test script:
Code: Select all
<?php
$ch = curl_init('http://www.google.com/finance?start=0&num=90&gl=us&hl=en&q=%28%28exchange%3ANYSE%29%20OR%20%28exchange%3ANASDAQ%29%20OR%20%28exchange%3AAMEX%29%29%20%5B%28MarketCap%20%3E%204250000000%20%7C%20MarketCap%20%3D%204250000000%29%20%26%20%28MarketCap%20%3C%202270000000000%20%7C%20MarketCap%20%3D%202270000000000%29%20%26%20%28PE%20%3E%200%20%7C%20PE%20%3D%200%29%20%26%20%28PE%20%3C%2034.18%20%7C%20PE%20%3D%2034.18%29%20%26%20%28Price13WeekPercChange%20%3E%20-16.39%20%7C%20Price13WeekPercChange%20%3D%20-16.39%29%20%26%20%28Price13WeekPercChange%20%3C%200.25%20%7C%20Price13WeekPercChange%20%3D%200.25%29%20%26%20%28QuoteLast%20%3E%200%20%7C%20QuoteLast%20%3D%200%29%20%26%20%28QuoteLast%20%3C%20115326%20%7C%20QuoteLast%20%3D%20115326%29%20%26%20%28QuotePercChange%20%3E%20-66.75%20%7C%20QuotePercChange%20%3D%20-66.75%29%20%26%20%28QuotePercChange%20%3C%2060.81%20%7C%20QuotePercChange%20%3D%2060.81%29%20%26%20%28AverageVolume%20%3E%201090000%20%7C%20AverageVolume%20%3D%201090000%29%20%26%20%28AverageVolume%20%3C%20802450000%20%7C%20AverageVolume%20%3D%20802450000%29%5D&restype=company&output=json&noIL=1');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$json = curl_exec($ch);
echo '<pre>';
echo 'Json result: '; var_dump(json_decode($json, true));
echo '</pre>';
switch(json_last_error()) {
case JSON_ERROR_DEPTH:
echo 'Maximum stack depth exceeded';
break;
case JSON_ERROR_CTRL_CHAR:
echo 'Unexpected control character found';
break;
case JSON_ERROR_SYNTAX:
echo 'Syntax error, malformed JSON';
break;
case JSON_ERROR_NONE:
echo 'No errors';
break;
}
?>
I got the following response
[text]Json result: NULL
Syntax error, malformed JSON[/text]
I would have posted the json string as well, however it is over double the size of our max allowed character count to post. But it appears there is something wrong with the syntax of the string, and not the depth.
How did you determine you were having a recursion issue?
How I determined it was a recursion issue is when I wrote the file I went in and deleted the lines
{
"start" : "0",
"num" : "90",
"num_company_results" : "99",
"num_mf_results" : "1",
"num_all_results" : "",
"original_query" : "((exchange:NYSE) OR (exchange:NASDAQ) OR (exchange:AMEX)) [(MarketCap \x3E 4250000000 | MarketCap = 4250000000) \x26 (MarketCap \x3C 2270000000000 | MarketCap = 2270000000000) \x26 (PE \x3E 0 | PE = 0) \x26 (PE \x3C 34.18 | PE = 34.18) \x26 (Price13WeekPercChange \x3E -16.39 | Price13WeekPercChange = -16.39) \x26 (Price13WeekPercChange \x3C 0.25 | Price13WeekPercChange = 0.25) \x26 (QuoteLast \x3E 0 | QuoteLast = 0) \x26 (QuoteLast \x3C 115326 | QuoteLast = 115326) \x26 (QuotePercChange \x3E -66.75 | QuotePercChange = -66.75) \x26 (QuotePercChange \x3C 60.81 | QuotePercChange = 60.81) \x26 (AverageVolume \x3E 1090000 | AverageVolume = 1090000) \x26 (AverageVolume \x3C 802450000 | AverageVolume = 802450000)]",
"query_for_display" : "((exchange:NYSE) OR (exchange:NASDAQ) OR (exchange:AMEX)) [(MarketCap > 4250000000 | MarketCap = 4250000000) & (MarketCap < 2270000000000 | MarketCap = 2270000000000) & (PE > 0 | PE = 0) & (PE < 34.18 | PE = 34.18) & (Price13WeekPercChange > -16.39 | Price13WeekPercChange = -16.39) & (Price13WeekPercChange < 0.25 | Price13WeekPercChange = 0.25) & (QuoteLast > 0 | QuoteLast = 0) & (QuoteLast < 115326 | QuoteLast = 115326) & (QuotePercChange > -66.75 | QuotePercChange = -66.75) & (QuotePercChange < 60.81 | QuotePercChange = 60.81) & (AverageVolume > 1090000 | AverageVolume = 1090000) & (AverageVolume < 802450000 | AverageVolume = 802450000)]",
"results_type" : "COMPANY",
"searchresults" :
and I deleted the last lines in the json file :
,
"mf_searchresults" :
[
]
}
And I only left the important json data. It still didn't work. Then, I removed all of the records, except for the first few. And it worked. After that I kept editing the size of the pertinent financial data. At certain sizes it works, and others no. Try removing those lines in the begging and end. It should be valid JSON. When benjamin said he decoded with no issue, I assumed everything was valid, and my problem was actually a memory one.
Re: last_json_error(), problem with JSON from Google
Posted: Thu May 20, 2010 6:38 pm
by Weirdan
Works for me:
Code: Select all
<?php
$ch = curl_init('http://www.google.com/finance?start=0&num=90&gl=us&hl=en&q=%28%28exchange%3ANYSE%29%20OR%20%28exchange%3ANASDAQ%29%20OR%20%28exchange%3AAMEX%29%29%20%5B%28MarketCap%20%3E%204250000000%20%7C%20MarketCap%20%3D%204250000000%29%20%26%20%28MarketCap%20%3C%202270000000000%20%7C%20MarketCap%20%3D%202270000000000%29%20%26%20%28PE%20%3E%200%20%7C%20PE%20%3D%200%29%20%26%20%28PE%20%3C%2034.18%20%7C%20PE%20%3D%2034.18%29%20%26%20%28Price13WeekPercChange%20%3E%20-16.39%20%7C%20Price13WeekPercChange%20%3D%20-16.39%29%20%26%20%28Price13WeekPercChange%20%3C%200.25%20%7C%20Price13WeekPercChange%20%3D%200.25%29%20%26%20%28QuoteLast%20%3E%200%20%7C%20QuoteLast%20%3D%200%29%20%26%20%28QuoteLast%20%3C%20115326%20%7C%20QuoteLast%20%3D%20115326%29%20%26%20%28QuotePercChange%20%3E%20-66.75%20%7C%20QuotePercChange%20%3D%20-66.75%29%20%26%20%28QuotePercChange%20%3C%2060.81%20%7C%20QuotePercChange%20%3D%2060.81%29%20%26%20%28AverageVolume%20%3E%201090000%20%7C%20AverageVolume%20%3D%201090000%29%20%26%20%28AverageVolume%20%3C%20802450000%20%7C%20AverageVolume%20%3D%20802450000%29%5D&restype=company&output=json&noIL=1');
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
$json = curl_exec($ch);
echo '<pre>';
echo 'Json result: '; var_dump(json_decode(preg_replace('#\\\\x[0-9a-fA-F]{2,2}#', '', $json), true));
echo '</pre>';
switch(json_last_error()) {
case JSON_ERROR_DEPTH:
echo 'Maximum stack depth exceeded';
break;
case JSON_ERROR_CTRL_CHAR:
echo 'Unexpected control character found';
break;
case JSON_ERROR_SYNTAX:
echo 'Syntax error, malformed JSON';
break;
case JSON_ERROR_NONE:
echo 'No errors';
break;
}
Re: last_json_error(), problem with JSON from Google
Posted: Thu May 20, 2010 6:56 pm
by zeus1
Weirdan, thats great news for me! What kind of setting do you have on your php server, like memory and recursion limits? What do you think could be causing mine to return null? My php settings are default.
Re: last_json_error(), problem with JSON from Google
Posted: Thu May 20, 2010 7:02 pm
by zeus1
Never mind. You did it! You cleaning of the JSON using
Code: Select all
preg_replace('#\\\\x[0-9a-fA-F]{2,2}#', '', $json)
fixed it. My original script works just fine! Cheers Weirdan!