Page 1 of 1

Problem with curl

Posted: Sun Apr 22, 2007 2:21 pm
by nadavvin
I want to make RSS to phpbb forum to a website.
Although it have RSS, It only contain the titles and not the content.

I try to do this:

Code: Select all

<?php

$ch = curl_init();
$post_data = array('search_keywords'=>'', 'search_terms'=>'any',
				   'search_author'=>"*", "search_forum"=>"-1",
	   "search_time"=>"1","search_fields"=>"all","search_cat"=>"-1",
	"sort_by"=>"0", "sort_dir"=>"DESC", "show_results"=>"posts", "return_chars"=>"1000");

curl_setopt($ch, CURLOPT_URL, "http://whatsup.org.il/index.php?name=PNphpBB2&file=search&mode=results" );

 curl_setopt($ch, CURLOPT_POST, 1 );

 curl_setopt($ch, CURLOPT_POSTFIELDS, $post_data);

curl_exec($ch);

 if (curl_errno($ch)) {
	 print curl_error($ch);
 }
 else echo "here";
curl_close($ch);

?>
but only here print.

What is the problem?

____________________________________________________________
I remove the line breaks in the array and now I get some output: "Empty reply from server"

What still is the problem?

__________________________________________________
I success with your forum to search after php.

I check the html code of whatsup and there is hidden field "sid" so I change the script to:

Code: Select all

<?php

$ch = curl_init();

$output=file_get_contents("http://whatsup.org.il/index.php?name=PNphpBB2&file=search");
preg_match('@input type="hidden" name="sid" value="([a-zA-Z1-9]+)@',$output,$matches);

$post_data = array('search_keywords'=>'', 'search_terms'=>'any', 'search_author'=>"*", "search_forum"=>"-1", "search_time"=>"1","search_fields"=>"all","search_cat"=>"-1", "sort_by"=>"0", "sort_dir"=>"DESC", "show_results"=>"posts", "return_chars"=>"1000", "sid"=>$matches[1]);
	
 curl_setopt($ch, CURLOPT_URL, "http://whatsup.org.il/index.php?name=PNphpBB2&file=search&mode=results" );

 curl_setopt($ch, CURLOPT_POST, 1 );

 curl_setopt($ch, CURLOPT_POSTFIELDS, $post_data);
curl_exec($ch);
 if (curl_errno($ch)) {
	 print curl_error($ch);
 }
 else echo "here";
curl_close($ch);

?>
but nothing see.

What's the problem again?

Posted: Mon Apr 23, 2007 7:34 am
by nadavvin
Is it possible that whatsup check the url which send the form?

Is there any curl option which change the referrer link?
____________________________________________________

I found "CURLOPT_URL"

but is still not work on the site.
The current code is:

Code: Select all

<?php

$ch = curl_init();
$post_data=Array
		(
		 'search_keywords' => '*',
		'search_terms' => 'any',
		'search_author' => '',
		'search_forum' => -1,
		'search_time' => 1,
		'search_fields' => 'all',
		'search_cat' => -1,
		'sort_by' => 0,
		'sort_dir' => 'DESC',
		'show_results' => 'posts',
		'return_chars' => 1000
		);
$link="http://whatsup.org.il/index.php?name=PNphpBB2&file=search&mode=results";

 curl_setopt($ch, CURLOPT_URL, $link );

 curl_setopt($ch, CURLOPT_POST, 1 );
curl_setopt($ch, CURLOPT_REFERER, $link);

 curl_setopt($ch, CURLOPT_POSTFIELDS, $post_data);
curl_exec($ch);
 if (curl_errno($ch)) {
	 print curl_error($ch);
 }
 else echo "here";
curl_close($ch);

?>
Can someone help me?

Posted: Mon Apr 23, 2007 9:47 am
by John Cartwright
It sounds like your scraping data without consent. :?

My help towards this solution will be as limited as you must pass a urlencoded string as the post vars, not an array. That is, until you can show me you have their consent to scrape content.

Posted: Mon Apr 23, 2007 10:35 am
by onion2k
Jcart wrote:My help towards this solution will be as limited as you must pass a urlencoded string as the post vars, not an array.
This is wrong. Curl expects an array for the CURLOPT_POSTFIELDS option.

Posted: Mon Apr 23, 2007 10:46 am
by John Cartwright
I was slightly wrong. Been awhile since I've used cURL.

You can pass either a url encoded string or an array.

Posted: Mon Apr 23, 2007 11:16 am
by nadavvin
Jcart wrote:It sounds like your scraping data without consent. :?

My help towards this solution will be as limited as you must pass a urlencoded string as the post vars, not an array. That is, until you can show me you have their consent to scrape content.
What the problem????

It like I search in their site but I want to create script which do it for me.