Page 1 of 1

search string parameters not working with curl?

Posted: Tue Sep 13, 2011 4:12 pm
by kc11
Hi All,

I am trying to submit a search string for google with php via curl through a proxy, http://instantaneoushidden.info/.

The search string is : http://www.google.com/search?q="dog"&nu ... r=all&Gf=0

It works fine. However when I try to submit ithe same string via curl, I get 10 results per page, not 100.

here is my curl function:

Code: Select all



<?php

function Curler($proxy, $inputstring, $ch) {
  

  $cookies = '..\cookies.txt'; 
  $timeout = 100; // MAX LENGTH CURL FUNCTION IN SECONDS BEFORE TIMEOUT
  $connecttimeout = 40; // CONNECTION TIMEOUT
  curl_setopt($ch,CURLOPT_URL,$proxy);
   curl_setopt($ch, CURLOPT_VERBOSE, 1);
  curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
 //curl_setopt($ch, CURLOPT_HEADER, 0);
 curl_setopt($ch, CURLOPT_COOKIEJAR, $cookies);   // Cookie management.
	curl_setopt($ch, CURLOPT_COOKIEFILE, $cookies);
  //curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Windows NT 5.1; rv:2.0.1) Gecko/20100101 Firefox/4.0.1');
	
		curl_setopt($ch, CURLOPT_ENCODING, 'gzip,deflate'); 
		curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); 
		curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true); 
                curl_setopt($ch, CURLOPT_TIMEOUT, $timeout);
    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $connecttimeout);
 
                  
            
    curl_setopt($ch,CURLOPT_POST,1);
  
 curl_setopt($ch, CURLOPT_POSTFIELDS, $inputstring);

 
                
    $response_1 = curl_exec($ch);  // actually make the request
    
   

 $info = curl_getinfo($ch);

 echo 'Took ' . $info['total_time'] . ' seconds to send a request to ' . $info['url'];
 var_dump($info);
    
    if(curl_exec($ch) === false)
{
    echo 'Curl error: ' . curl_error($ch);
    return false;
}
else
{
    echo 'Operation completed without any errors';
    return $response_1;
}

        
}

?>



Does anyone have any ideas? I'm stumped .

Thanks,

KC

Re: search string parameters not working with curl?

Posted: Wed Sep 14, 2011 1:43 am
by greip
I see you have commented out setting the UA-string. Have you tried running the script with the UA-string set to something other that the default? Google does not like web scrapers, and might classify you as a scraper if the UA-string reveals that the client is cURL.

Re: search string parameters not working with curl?

Posted: Wed Sep 21, 2011 8:55 am
by kc11
Thanks for the response Geir,

It turned out that I didn't urlencode the string. Its now working.

Regards,

KC