Page 1 of 1

Help with curl

Posted: Sun Oct 12, 2008 11:04 pm
by ngmix
Hello all, I have a yahoogroups message board.
For effectiveness, I need to move some of the messages to my private board on my website.

When I use Curl to download the message, I get this phrase:
Aren't you supposed to be somewhere else?

Has anyone done this before,
please assist! how can I get it done.

Thank you.

Re: Help with curl

Posted: Mon Oct 13, 2008 6:51 am
by onion2k
By the sounds of it Yahoo's site is expecting the correct cookie/session/referer/a combination of all three, and you're not sending it. Basically you'll need to write something that follows the entire process from logging in to fetching the message as a browser would, but in PHP.

Re: Help with curl

Posted: Mon Oct 13, 2008 9:19 am
by ngmix
thanks for the response, this is my code here:

// I have created a file for cookiefile, one for cookiejar, and one for sample.txt and they are all world readable and writeable
//after all check has been done, download the new id's file - Im going to use curl.
$file = "sample.txt";
$url = "http://groups.yahoo.com/group/NaijaPoli ... age/138888";
echo cURLdownload( $url, $file);

function cURLdownload($url, $file)
{
//if( !cURLcheckBasicFunctions() ) return "UNAVAILABLE: cURL Basic Functions";
$ch = curl_init();
if($ch)
{
$fp = fopen($file, "w");
if($fp)
{
if( !curl_setopt($ch, CURLOPT_URL, $url) ) return "FAIL: curl_setopt(CURLOPT_URL)";
if( !curl_setopt($ch, CURLOPT_FILE, $fp) ) return "FAIL: curl_setopt(CURLOPT_FILE)";
if( !curl_setopt($ch, CURLOPT_HEADER, 0) ) return "FAIL: curl_setopt(CURLOPT_HEADER)";
if( !curl_setopt($ch, CURLOPT_COOKIEFILE, './cukiefile') ) return "FAIL: curl_setopt(CURLOPT_cookiefile)";
if( !curl_setopt($ch, CURLOPT_COOKIEJAR, './cukiejar') ) return "FAIL: curl_setopt(CURLOPT_cookiejar)";
if( !curl_exec($ch) ) return "FAIL: curl_exec()";
curl_close($ch);
fclose($fp);
return "SUCCESS: $file [$url]";
}
else return "FAIL: fopen()";
}
else return "FAIL: curl_init()";
}

After all this, I get a response in my sample.txt that says Aren't you supposed to be somewhere else?
I know that I am doing something wrong with the cookie/session setting, I just don't know what it is.
I'ld appreciate it if anyone can help.

Thanks again.

Re: Help with curl

Posted: Mon Oct 13, 2008 9:36 am
by onion2k
If I go to that Yahoo page it redirects me to the login page. I think that's pretty much certain that you'll need to make your PHP script log into Yahoo before it can access the right pages. You'll need to examine the login form to figure out what to send...

Re: Help with curl

Posted: Sun Oct 19, 2008 10:33 am
by ngmix
Thanks again,

but whenever I try running the program, my machine is already logged into yahoo where I am already a registered user of the forum. When I type the address into my address bar, I am able to see the page. Just doesn't work when I try to download using the program.

Re: Help with curl

Posted: Sun Oct 19, 2008 11:01 am
by onion2k
ngmix wrote:but whenever I try running the program, my machine is already logged into yahoo where I am already a registered user of the forum. When I type the address into my address bar, I am able to see the page. Just doesn't work when I try to download using the program.
Whichever browser you've used is logged into yahoo. Your machine isn't. For example, if you log into Yahoo with Firefox, then look at the page with IE7, IE7 won't be logged in. It's because each session is unique to each browser.

PHP with Curl is essentially another browser in that respect. There is no way to cheat*. You'll have to write the code to log into Yahoo with PHP. It's going to be a lot of work.

* Actually there is. You could copy the value of the Yahoo cookies when you log in using an ordinary browser into Curl's cookiejar file in PHP. It'd be a little unreliable though, and Yahoo probably check the user agent to block that anyway.

Re: Help with curl

Posted: Tue Oct 21, 2008 6:42 pm
by ngmix
onion2k wrote:
* Actually there is. You could copy the value of the Yahoo cookies when you log in using an ordinary browser into Curl's cookiejar file in PHP. It'd be a little unreliable though, and Yahoo probably check the user agent to block that anyway.
I tried copying the cookie into a textfile, I used cookiejar and cookiefile as shown in the code, to store the cookies. I think yahoo blocked that already. I appreciate you helping me along.
How can I write my own code to log into yahoo. If I can get a guide, I don't mind spending the time to write it. I'm well versed with php, if you can just point me in the right direction.

Thanks a lot.