Alternate solution to file_get_contents() on a remote file

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
jonaphin
Forum Newbie
Posts: 10
Joined: Sun May 07, 2006 2:47 pm

Alternate solution to file_get_contents() on a remote file

Post by jonaphin »

Hey Guys,
I've been busting my brains out trying to find a solution to that problem. I need to save the output of a file located on a different server from the php script. On some servers, you could just do file_get_contents("http://thatwebsite.com/webpage.ext") and save the output in a variable. In my case, I do not have the read file permissions on a remote server, even with allow_url_fopen set to "On". So, I've tried to use cURL to no avail. Any help in the matter is appreciated.

Thx
Jonaphin
User avatar
s.dot
Tranquility In Moderation
Posts: 5001
Joined: Sun Feb 06, 2005 7:18 pm
Location: Indiana

Post by s.dot »

If you don't have read permission on the file, you won't be able to access it in any way.
If you have access to the remote server you change the permissions on it.
If not, you're pretty much stuck :P
Set Search Time - A google chrome extension. When you search only results from the past year (or set time period) are displayed. Helps tremendously when using new technologies to avoid outdated results.
jonaphin
Forum Newbie
Posts: 10
Joined: Sun May 07, 2006 2:47 pm

Post by jonaphin »

Which leads me to this question... How come am i able to access remote files via cURL on certain websites, but not others?

Cross out specifying the REFERER as an answer... it doesn't seem to have something to do with that.
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

Likely, those sites deny the request. It may have little to do with your server and settings, but we can't tell you for sure without seeing the code.
jonaphin
Forum Newbie
Posts: 10
Joined: Sun May 07, 2006 2:47 pm

Post by jonaphin »

feyd | Please use

Code: Select all

,

Code: Select all

and [syntax="..."] tags where appropriate when posting code. Your post has been edited to reflect how we'd like it posted. Please read:  [url=http://forums.devnetwork.net/viewtopic.php?t=21171]Posting Code in the Forums[/url] to learn how to do it too.[/color]


Here is the code:

Code: Select all

if(!empty($_POST["submit"])) {
$url = $_POST["url"];
echo "output: ".getCode($url);
}

function getCode($url) {
// create a new curl resource
$ch = curl_init();

// set URL and options
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);

// Tried with fake referrer
//curl_setopt($ch, CURLOPT_REFERER, "http://localsearchmaps.com");

// verbose for debugging purposes
curl_setopt($ch, CURLOPT_VERBOSE, 1);


// grab URL
$output = curl_exec($ch);

// Print info
echo '<pre>';
print_r (curl_geti! nfo($ch));
echo '</pre>';

curl_close($ch);
return $output;
}
?>

<!-- form that helps you test different urls -->
<form action="<?=$_SERVER["php_self"];?>" method="post">
url: <input type="text" name="url" value="<?=$url;?>" />
<input type="submit" name="submit" value="submit" />
</form>

Well, what's odd is that this code will work fine with some sites like the payment gateway http://authorize.net or the roommate finder http://roompals.com
but not with that url above, localsearchmaps.com.

Hoping someone can help.
Jonaphin


feyd | Please use

Code: Select all

,

Code: Select all

and [syntax="..."] tags where appropriate when posting code. Your post has been edited to reflect how we'd like it posted. Please read:  [url=http://forums.devnetwork.net/viewtopic.php?t=21171]Posting Code in the Forums[/url] to learn how to do it too.[/color]
jonaphin
Forum Newbie
Posts: 10
Joined: Sun May 07, 2006 2:47 pm

Thx

Post by jonaphin »

Thank you for editing, it looks much nicer now... I'm a noob in bbcode, ain't that ironic for a programmer?
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

Some servers deny requests to user agents they don't recognise (bot protection).

You need to "spoof" the user agent string. Does the owner of this content that appears to be protected in this fashion know what you're doing? Are you planning upon displaying this information on your own web pages?
jonaphin
Forum Newbie
Posts: 10
Joined: Sun May 07, 2006 2:47 pm

Post by jonaphin »

The site owner knows it's going to be used, and provides the service free ( provided you do not abuse it ), but not necessarily using cURL or f_get_contents... It might be a judicious idea talking to him about it.

Thanks for your answer, it seems like a very reasonable one. I had originally thought about "REFERRER" spoofing, but it might just be user agent I should have looked into. I'll keep you posted on the matter :)

Thanks a bunch,
Jonaphin
jonaphin
Forum Newbie
Posts: 10
Joined: Sun May 07, 2006 2:47 pm

Post by jonaphin »

I guess it's now solved. Many hosting companies do not readily allow remote socket/non-socket connections. Which is where all the trouble comes from. :cry:
Post Reply