unable to get webpage contents of a URL in PHP

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
pepperboy123
Forum Newbie
Posts: 2
Joined: Fri May 06, 2016 3:13 pm

unable to get webpage contents of a URL in PHP

Post by pepperboy123 »

Hi,

I am trying to open a webpage and get it's contents. On the browser, it works fine. However, through PHP, I'm unable to get webpage contents. In fact, I'm not able to hit the URL either. I feel anything apart from a header is not able to discover the webpage. With other URLs, it works fine. Please suggest me a method to able to get the contents of this webpage.

Code: Select all

URL : https://kat.cr/usearch/life%20of%20pi/
My code attempts

1. fopen

Code: Select all

$url = "https://kat.cr/usearch/life%20of%20pi/";
    $handle = fopen($url, "r");
2. Using curl

Code: Select all

/* gets the data from a URL */
function get_data($url) {
    $userAgent = 'Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; .NET CLR 1.1.4322)';
    $ch = curl_init();
    $timeout = 5;
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_HEADER, 0);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_ENCODING, 'gzip');
    curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
    $data = curl_exec($ch);
    curl_close($ch);
    return $data;
}
$url = "https://kat.cr/usearch/life%20of%20pi/";
$returned_content = get_data($url);
    echo $returned_content;

}
PS : In all of the above methods, I also tried prepending the URL with compress.zlib since it initially gave me encrypted characters. So the URL would look like the following. Still no luck.

Code: Select all

$url = "compress.zlib://https://kat.cr/usearch/life%20of%20pi/";
PS : I had posted a similar question on another forum, however, I wasn't sure whether it was published due to site problems, and am not tracking it. My intention is not to cross post :)
User avatar
Celauran
Moderator
Posts: 6427
Joined: Tue Nov 09, 2010 2:39 pm
Location: Montreal, Canada

Re: unable to get webpage contents of a URL in PHP

Post by Celauran »

pepperboy123 wrote:However, through PHP, I'm unable to get webpage contents.
OK. What happens instead? What sort of error(s) are you seeing?
pepperboy123
Forum Newbie
Posts: 2
Joined: Fri May 06, 2016 3:13 pm

Re: unable to get webpage contents of a URL in PHP

Post by pepperboy123 »

Celauran,
I tried quite a few times to ensure my reply is useful. I don't see any errors, but for the same debug code that I wrote just now, I sometimes get encrypted characters, and some times, the page just keeps spinning for an awefully long time, and does not return anything.

Why don't you try executing this debug code? What I've done is I'm opening the URLs in the array - the 1st and 3rd links are the ones which keep spinning, the 2nd and the 4th do not give any problem

Even, if I get encrypted characters constantly, it's a good thing, so I can decompress it using some method, may be prepend "compress.zlip://" before the URL, but if it just keeps spinning, not a good thing :)

Code: Select all

<?php

$statusArray = array();
$allURLs = array("https://kat.cr/usearch/life%20of%20pi/","https://en.wikipedia.org/wiki/Basketball","https://kat.cr/usearch/life%20of%20pi/","https://en.wikipedia.org/wiki/Basketball");

for($i = 0;$i < count($allURLs) ; $i++)
{
    $handle = fopen($allURLs[$i], "r");
    if ($handle)
    {
        // error opening the file.
        array_push($statusArray,"able to open the page!");
    }
    else
    {
        // error opening the file.
        array_push($statusArray,"404");
    }

    /* printing first line of page now */
    $line = fgets($handle);
    echo $line;

    fclose($handle);
}

print_r($statusArray);
?>
thinsoldier
Forum Contributor
Posts: 367
Joined: Fri Jul 20, 2007 11:29 am
Contact:

Re: unable to get webpage contents of a URL in PHP

Post by thinsoldier »

[$]> php testweb.php
vS`4WHgt&J$&);g[%KhS(D
^[[?6c<!DOCTYPE html>
v):H@,^cO)x{D
JVfgGvH@Y啈4~mt46Nqqq!^bmۍOF\/:9$q|BcaN/FIzFc'ir4KUkܔrڹG/a_xn2>ryJK<YjdEf.7M0('8K"7$G~dFdQ*'y.P B/pT|ziEAL@~7w<9S
^[[?6c<!DOCTYPE html>
Array
(
[0] => able to open the page!
[1] => able to open the page!
[2] => able to open the page!
[3] => able to open the page!
)

print_r says I could open all urls.
Warning: I have no idea what I'm talking about.
thinsoldier
Forum Contributor
Posts: 367
Joined: Fri Jul 20, 2007 11:29 am
Contact:

Re: unable to get webpage contents of a URL in PHP

Post by thinsoldier »

Off topic: why is it in the last year or so I've seen so many people using regular for-loops instead of foreach? I've literally gone years at a time without having a need for a regular for loop in php. Why the seemingly sudden increase in popularity for for-loops and array_push instead of regular array appending sytax?

example: https://gist.github.com/thinsoldier/b84 ... 8b5e437791
Warning: I have no idea what I'm talking about.
Post Reply