Page 1 of 1

Alternate solution to file_get_contents() on a remote file

Posted: Sun May 07, 2006 3:07 pm
by jonaphin
Hey Guys,
I've been busting my brains out trying to find a solution to that problem. I need to save the output of a file located on a different server from the php script. On some servers, you could just do file_get_contents("http://thatwebsite.com/webpage.ext") and save the output in a variable. In my case, I do not have the read file permissions on a remote server, even with allow_url_fopen set to "On". So, I've tried to use cURL to no avail. Any help in the matter is appreciated.

Thx
Jonaphin

Posted: Sun May 07, 2006 3:36 pm
by s.dot
If you don't have read permission on the file, you won't be able to access it in any way.
If you have access to the remote server you change the permissions on it.
If not, you're pretty much stuck :P

Posted: Sun May 07, 2006 4:32 pm
by jonaphin
Which leads me to this question... How come am i able to access remote files via cURL on certain websites, but not others?

Cross out specifying the REFERER as an answer... it doesn't seem to have something to do with that.

Posted: Sun May 07, 2006 4:48 pm
by feyd
Likely, those sites deny the request. It may have little to do with your server and settings, but we can't tell you for sure without seeing the code.

Posted: Sun May 07, 2006 7:05 pm
by jonaphin
feyd | Please use

Code: Select all

,

Code: Select all

and [syntax="..."] tags where appropriate when posting code. Your post has been edited to reflect how we'd like it posted. Please read:  [url=http://forums.devnetwork.net/viewtopic.php?t=21171]Posting Code in the Forums[/url] to learn how to do it too.[/color]


Here is the code:

Code: Select all

if(!empty($_POST["submit"])) {
$url = $_POST["url"];
echo "output: ".getCode($url);
}

function getCode($url) {
// create a new curl resource
$ch = curl_init();

// set URL and options
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

// Tried with fake referrer
//curl_setopt($ch, CURLOPT_REFERER, "http://localsearchmaps.com");

// verbose for debugging purposes
curl_setopt($ch, CURLOPT_VERBOSE, 1);


// grab URL
$output = curl_exec($ch);

// Print info
echo '<pre>';
print_r (curl_geti! nfo($ch));
echo '</pre>';

curl_close($ch);
return $output;
}
?>

<!-- form that helps you test different urls -->
<form action="<?=$_SERVER["php_self"];?>" method="post">
url: <input type="text" name="url" value="<?=$url;?>" />
<input type="submit" name="submit" value="submit" />
</form>

Well, what's odd is that this code will work fine with some sites like the payment gateway http://authorize.net or the roommate finder http://roompals.com
but not with that url above, localsearchmaps.com.

Hoping someone can help.
Jonaphin


feyd | Please use

Code: Select all

,

Code: Select all

and [syntax="..."] tags where appropriate when posting code. Your post has been edited to reflect how we'd like it posted. Please read:  [url=http://forums.devnetwork.net/viewtopic.php?t=21171]Posting Code in the Forums[/url] to learn how to do it too.[/color]

Thx

Posted: Mon May 08, 2006 9:24 am
by jonaphin
Thank you for editing, it looks much nicer now... I'm a noob in bbcode, ain't that ironic for a programmer?

Posted: Mon May 08, 2006 9:53 am
by Chris Corbyn
Some servers deny requests to user agents they don't recognise (bot protection).

You need to "spoof" the user agent string. Does the owner of this content that appears to be protected in this fashion know what you're doing? Are you planning upon displaying this information on your own web pages?

Posted: Mon May 08, 2006 11:05 am
by jonaphin
The site owner knows it's going to be used, and provides the service free ( provided you do not abuse it ), but not necessarily using cURL or f_get_contents... It might be a judicious idea talking to him about it.

Thanks for your answer, it seems like a very reasonable one. I had originally thought about "REFERRER" spoofing, but it might just be user agent I should have looked into. I'll keep you posted on the matter :)

Thanks a bunch,
Jonaphin

Posted: Wed May 10, 2006 7:47 am
by jonaphin
I guess it's now solved. Many hosting companies do not readily allow remote socket/non-socket connections. Which is where all the trouble comes from. :cry: