Page 1 of 1

time out problem in curl

Posted: Sun Aug 03, 2008 7:01 am
by bhavin12300
hi
i am curl to fetch webpage from server.
i want to fetch two webpage one after another.

when i request one page from script,it is working perfectly.but when i insert code to fetch another page after first page it give me timeout error.
it worked perfectly fine if i request both page in individual script,but when i insert request code for file "two" after request code of file "one", it give me timeout error.
Code:

Code: Select all

$web_page = http_get("http://travel.travelocity.com/flights/AirSearch.do?SEQ=1217741147969732008&breadcrumbStart=1", $referer="");
$web_page = http_get("http://travel.travelocity.com/flights/AirSearch.do?SEQ=1217741147969732008&breadcrumbStart=2", $referer="");
These are two pages.
Again,
if i insert one of any request in script alone it is working just fine but not working if i put both together.
whats wrong in it???

Re: time out problem in curl

Posted: Sun Aug 03, 2008 7:36 am
by ody3307
This doesn't look like you are using cURL. You don't have to but you mentioned it in you post.

If you just want to grab these 2 pages, and do something with the results afterwards, you'll need to append the second page to the variable. As you have it written now, the second call overwrites the result of the first call.

So, if you want both pages in your variable, change $web_page = TO $web_page .= on the second call.

If you want to grab a page, do something with it, then grab another page, then put your code to "do something" between these 2 calls.

EDITED:
Oops forgot about changing timeout.

Use set_time_limit() to change the maximum time limit for the script.

Re: time out problem in curl

Posted: Sun Aug 03, 2008 11:48 am
by bhavin12300
sir,
http_get is my function as its definition is as follow.

Code: Select all

function http_get($target, $ref)
    {
    return http($target, $ref, $method="GET", $data_array="", EXCL_HEAD);
    }
function http($target, $ref, $method, $data_array, $incl_head)
    {
    # Initialize PHP/CURL handle
    $ch = curl_init();
 
    # Prcess data, if presented
    if(is_array($data_array))
        {
        # Convert data array into a query string (ie animal=dog&sport=baseball)
        foreach ($data_array as $key => $value) 
            {
            if(strlen(trim($value))>0)
                $temp_string[] = $key . "=" . urlencode($value);
            else
                $temp_string[] = $key;
            }
        $query_string = join('&', $temp_string);
        }
        
    # HEAD method configuration
    if($method == HEAD)
        {
        curl_setopt($ch, CURLOPT_HEADER, TRUE);                // No http head
        curl_setopt($ch, CURLOPT_NOBODY, TRUE);                // Return body
        }
    else
        {
        # GET method configuration
        if($method == GET)
            {
            if(isset($query_string))
                $target = $target . "?" . $query_string;
            curl_setopt ($ch, CURLOPT_HTTPGET, TRUE); 
            curl_setopt ($ch, CURLOPT_POST, FALSE); 
            }
        # POST method configuration
        if($method == POST)
            {
            if(isset($query_string))
                curl_setopt ($ch, CURLOPT_POSTFIELDS, $query_string);
            curl_setopt ($ch, CURLOPT_POST, TRUE); 
            curl_setopt ($ch, CURLOPT_HTTPGET, FALSE); 
            }
        curl_setopt($ch, CURLOPT_HEADER, $incl_head);   // Include head as needed
        curl_setopt($ch, CURLOPT_NOBODY, FALSE);        // Return body
        }
        
    curl_setopt($ch, CURLOPT_COOKIEJAR, COOKIE_FILE);   // Cookie management.
    curl_setopt($ch, CURLOPT_COOKIEFILE, COOKIE_FILE);
    curl_setopt($ch, CURLOPT_TIMEOUT, CURL_TIMEOUT);    // Timeout
    curl_setopt($ch, CURLOPT_USERAGENT, WEBBOT_NAME);   // Webbot name
    curl_setopt($ch, CURLOPT_URL, $target);             // Target site
    curl_setopt($ch, CURLOPT_REFERER, $ref);            // Referer value
    curl_setopt($ch, CURLOPT_VERBOSE, FALSE);           // Minimize logs
    curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, FALSE);    // No certificate
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);     // Follow redirects
    curl_setopt($ch, CURLOPT_MAXREDIRS, 4);             // Limit redirections to four
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);     // Return in string
    
    # Create return array
    $return_array['FILE']   = curl_exec($ch); 
    $return_array['STATUS'] = curl_getinfo($ch);
    $return_array['ERROR']  = curl_error($ch);
    
    # Close PHP/CURL handle
    curl_close($ch);
    
    # Return results
    return $return_array;
    }
 
 
and by the way i dont want to append one page with other .what i want to do is i want to grab first page and doing something on it and than want to grab second page.
but when i do that timer exceed.

Re: time out problem in curl

Posted: Sun Aug 03, 2008 1:22 pm
by ody3307
Okay, it looks like you are using Michael Shrenck's code library. I'm pretty familiar with it. I don't see anything obvious so far. Can we see the code where you request the first page, then the second page?

Thanks

Re: time out problem in curl

Posted: Mon Aug 04, 2008 4:46 am
by bhavin12300

Code: Select all

 
$web_page = http_get("http://travel.travelocity.com/flights/InitialSearch.do?dateLeavingTime=Anytime&dateReturningTime=Anytime&seniors=0&children=0&flightType=roundtrip&dateReturningMonth=Aug&dateReturningDay=23&adults=1&leavingFrom=PDX&dateLeavingMonth=Aug&goingTo=SJC&dateLeavingDay=20", $referer="");
 
  
   $web_page = http_get("http://travel.travelocity.com/flights/ResolveAirportAction.do?".$seq, $referer="");
  
 
 
 
 $web_page  = http_get("http://travel.travelocity.com/flights/AirSearch.do?".$seq."&breadcrumbStart=1", $referer="");
 $web_page = http_get("http://travel.travelocity.com/flights/AirSearch.do?".$seq."&breadcrumbStart=2", $referer="");
echo $web_page['FILE'];
In these last two line i am getting time exceed error.if i remove any one from it than it worked perfectly nice.