Reading a webpage (using cURL) => How?
Posted: Wed Jul 14, 2004 4:06 am
Hello,
I would like to grab text of a given URL and place it in a string.
I got involved in using cURL and would like to "leach" a website. Since cURL offers a lot of possibilities to set a timeout and delivers detailed error messages and error numbers, it seems to be a good choice.
Following PHP script reads a website. This actually works fine, but there is one problem, the output dosn't show pictures and links of relative pictures and links. E.g. instead of http://www.url.de/folder/test.gif my browser output shows http://192.xxx.x.xxx/folder/test.gif (the IP of my internal webserver)
=> O.k. this of course just concerns relative links. So my question:
How can I use cURL and make the server I am requesting think I am a regular browser client and not as a server requesting. I set CURLOPT_USERAGENT to "Mozilla/4.0", unfortunately this does not help.
PHP source code:
I would appreciate any help!
Thanks,
visionmaster
feyd | Please use
I would like to grab text of a given URL and place it in a string.
I got involved in using cURL and would like to "leach" a website. Since cURL offers a lot of possibilities to set a timeout and delivers detailed error messages and error numbers, it seems to be a good choice.
Following PHP script reads a website. This actually works fine, but there is one problem, the output dosn't show pictures and links of relative pictures and links. E.g. instead of http://www.url.de/folder/test.gif my browser output shows http://192.xxx.x.xxx/folder/test.gif (the IP of my internal webserver)
=> O.k. this of course just concerns relative links. So my question:
How can I use cURL and make the server I am requesting think I am a regular browser client and not as a server requesting. I set CURLOPT_USERAGENT to "Mozilla/4.0", unfortunately this does not help.
PHP source code:
Code: Select all
<?php
$string = download("http://www.url.de");
echo $string;
function download($url) {
$ch = curl_init($url);
curl_setopt ($ch, CURLOPT_URL, $url);
curl_setopt ($ch, CURLOPT_HEADER, 0);
curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt ($ch, CURLOPT_USERAGENT, "Mozilla/4.0");
$result = curl_exec ($ch);
curl_close ($ch);
return $result;
}
?>Thanks,
visionmaster
feyd | Please use
Code: Select all
tags when posting code. Read: [url=http://forums.devnetwork.net/viewtopic.php?t=21171]Posting Code in the Forums[/url][/color]