Page 1 of 1
Can Curl be blocked?
Posted: Tue Jul 27, 2010 8:45 am
by Rox23
I was wondering if there was a way to block Curl POST requests?
I have a website that I am trying to use Curl with and no matter what I try, the page does NOT recognize the POST fields. The website was originally done by someone else. I have gone through the code and don't see anything that could do this?
I tried pointing the Curl POST code to another server and it all works. So my belief is there must be something blocking the curl request.
Any ideas?
Thanks!
Re: Can Curl be blocked?
Posted: Tue Jul 27, 2010 8:47 am
by Benjamin
Maybe the site is sending a header redirect because there are no cookies set, or you are not logged in.
Re: Can Curl be blocked?
Posted: Tue Jul 27, 2010 10:01 am
by Rox23
Benjamin wrote:Maybe the site is sending a header redirect because there are no cookies set, or you are not logged in.
I thought of that and I set cookies in the curl script.
I have tried using several diferent user agents and tried using "1" instead of "true". The results are always the same.
I have tried posting the fields as an array and as a concatenated string - no change.
Code: Select all
$target_url = $webs['website_url'];
$userAgent = 'Googlebot/2.1 (http://www.googlebot.com/bot.html)';
$fields = array();
$fields_string = "";
$ch = curl_init();
curl_setopt($ch, CURLOPT_USERAGENT,$_SERVER['HTTP_USER_AGENT']);
curl_setopt($ch, CURLOPT_URL,$target_url);
foreach($fields as $key=>$value) { $fields_string .= $key.'='.$value.'&'; }
$fields_string = rtrim($fields_string,'&');
curl_setopt($ch, CURLOPT_POST,1);
curl_setopt($ch, CURLOPT_POSTFIELDS,$fields_string);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_HEADER ,false );
curl_setopt($ch, CURLOPT_TIMEOUT, 1000);
curl_setopt($ch, CURLOPT_COOKIEJAR, 'cookie.txt');
curl_setopt($ch, CURLOPT_COOKIEFILE, 'cookie.txt');
$html= curl_exec($ch);
Re: Can Curl be blocked?
Posted: Tue Jul 27, 2010 10:06 am
by Benjamin
Well, google doesn't send POST requests, so I'm not sure why you would be using that user agent. Are you sure this is your site? Have you checked to see what is being returned?
Re: Can Curl be blocked?
Posted: Tue Jul 27, 2010 2:03 pm
by Rox23
I found that user agent code online somewhere.
I am just trying anything that could make sense to work. I have tried browser user agents (mozilla, netscape, etc...) and nothing.
I tried removing all the cURL options except the POST stuff and URL. Still won't work.
I get no post field results. It is as if the POST never happened.
Is there code that can block this type of POST that I need to look for in this website or server configuration?
Thanks!