Page 1 of 1

unable to get webpage contents of a URL in PHP

Posted: Fri May 06, 2016 3:18 pm
by pepperboy123
Hi,

I am trying to open a webpage and get it's contents. On the browser, it works fine. However, through PHP, I'm unable to get webpage contents. In fact, I'm not able to hit the URL either. I feel anything apart from a header is not able to discover the webpage. With other URLs, it works fine. Please suggest me a method to able to get the contents of this webpage.

Code: Select all

URL : https://kat.cr/usearch/life%20of%20pi/
My code attempts

1. fopen

Code: Select all

$url = "https://kat.cr/usearch/life%20of%20pi/";
    $handle = fopen($url, "r");
2. Using curl

Code: Select all

/* gets the data from a URL */
function get_data($url) {
    $userAgent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)';
    $ch = curl_init();
    $timeout = 5;
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_HEADER, 0);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_ENCODING, 'gzip');
    curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
    $data = curl_exec($ch);
    curl_close($ch);
    return $data;
}
$url = "https://kat.cr/usearch/life%20of%20pi/";
$returned_content = get_data($url);
    echo $returned_content;

}
PS : In all of the above methods, I also tried prepending the URL with compress.zlib since it initially gave me encrypted characters. So the URL would look like the following. Still no luck.

Code: Select all

$url = "compress.zlib://https://kat.cr/usearch/life%20of%20pi/";
PS : I had posted a similar question on another forum, however, I wasn't sure whether it was published due to site problems, and am not tracking it. My intention is not to cross post :)

Re: unable to get webpage contents of a URL in PHP

Posted: Fri May 06, 2016 4:42 pm
by Celauran
pepperboy123 wrote:However, through PHP, I'm unable to get webpage contents.
OK. What happens instead? What sort of error(s) are you seeing?

Re: unable to get webpage contents of a URL in PHP

Posted: Sat May 07, 2016 1:36 am
by pepperboy123
Celauran,
I tried quite a few times to ensure my reply is useful. I don't see any errors, but for the same debug code that I wrote just now, I sometimes get encrypted characters, and some times, the page just keeps spinning for an awefully long time, and does not return anything.

Why don't you try executing this debug code? What I've done is I'm opening the URLs in the array - the 1st and 3rd links are the ones which keep spinning, the 2nd and the 4th do not give any problem

Even, if I get encrypted characters constantly, it's a good thing, so I can decompress it using some method, may be prepend "compress.zlip://" before the URL, but if it just keeps spinning, not a good thing :)

Code: Select all

<?php

$statusArray = array();
$allURLs = array("https://kat.cr/usearch/life%20of%20pi/","https://en.wikipedia.org/wiki/Basketball","https://kat.cr/usearch/life%20of%20pi/","https://en.wikipedia.org/wiki/Basketball");

for($i = 0;$i < count($allURLs) ; $i++)
{
    $handle = fopen($allURLs[$i], "r");
    if ($handle)
    {
        // error opening the file.
        array_push($statusArray,"able to open the page!");
    }
    else
    {
        // error opening the file.
        array_push($statusArray,"404");
    }

    /* printing first line of page now */
    $line = fgets($handle);
    echo $line;

    fclose($handle);
}

print_r($statusArray);
?>

Re: unable to get webpage contents of a URL in PHP

Posted: Wed May 11, 2016 2:31 pm
by thinsoldier
[$]> php testweb.php
vS`4WHgt&J$&);g[%KhS(D
^[[?6c<!DOCTYPE html>
v):H@,^cO)x{D
JVfgGvH@Y啈4~mt46Nqqq!^bmۍOF\/:9$q|BcaN/FIzFc'ir4KUkܔrڹG/a_xn2>ryJK<YjdEf.7M0('8K"7$G~dFdQ*'y.P B/pT|ziEAL@~7w<9S
^[[?6c<!DOCTYPE html>
Array
(
[0] => able to open the page!
[1] => able to open the page!
[2] => able to open the page!
[3] => able to open the page!
)

print_r says I could open all urls.

Re: unable to get webpage contents of a URL in PHP

Posted: Wed May 11, 2016 2:48 pm
by thinsoldier
Off topic: why is it in the last year or so I've seen so many people using regular for-loops instead of foreach? I've literally gone years at a time without having a need for a regular for loop in php. Why the seemingly sudden increase in popularity for for-loops and array_push instead of regular array appending sytax?

example: https://gist.github.com/thinsoldier/b84 ... 8b5e437791