php cURL problem

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
fredass
Forum Newbie
Posts: 12
Joined: Tue Apr 06, 2010 4:18 pm

php cURL problem

Post by fredass »

Hi,
I need to download a page source code to server and save it in the html file. First, I've tried to validate that page in http://validator.w3.org/ and got the answer "500 Internal Server Error"... later I've tried to download source code with fopen() function and it returned "failed to open stream: HTTP request failed! HTTP/1.1 500 Internal Server Error". The syntax was correct because that code worked with other URLs. Then I've tried to use cURL. The code is:

Code: Select all

<?php
	$ch = curl_init("https://pirkimai.eviesiejipirkimai.lt/app/notice/notices.asp?B=PPO");

	$fp = fopen("./system/application/views/downloaded_html/home.html", "w");

	curl_setopt($ch, CURLOPT_FILE, $fp);
	curl_setopt($ch, CURLOPT_HEADER, 0);

	curl_exec($ch);
	curl_close($ch);
	fclose($fp);
	
	echo "done";	
?>
The result was a source code of only six lines when real source code of that page is much bigger. I can see that code using browser's "view source code" function. The first thought was that they are preventing connections from other servers or something.. or can it be that curl_init() function cant understand the given link? maybe because of the question mark? As you can see the URL of that page is https://pirkimai.eviesiejipirkimai.lt/a ... .asp?B=PPO. And also I've tested this code with other URLs ant it worked just fine. Any ideas? I woud appreciate any help. Thanks!
User avatar
requinix
Spammer :|
Posts: 6617
Joined: Wed Oct 15, 2008 2:35 am
Location: WA, USA

Re: php cURL problem

Post by requinix »

First of all, don't overcomplicate the issue: just use copy.

Code: Select all

copy("https://pirkimai.eviesiejipirkimai.lt/app/notice/notices.asp?B=PPO",
     "./system/application/views/downloaded_html/home.html");
Are you getting a 500 on their server or on yours? If yours, check the server logs for what the problem is.
fredass
Forum Newbie
Posts: 12
Joined: Tue Apr 06, 2010 4:18 pm

Re: php cURL problem

Post by fredass »

ok, I've changed it to copy() and still get "failed to open stream: HTTP request failed! HTTP/1.1 500 Internal Server Error ". The only log file that has changed was "access log" with message: "78.158.***.*** - - [07/Apr/2010:10:06:04 +0300] "GET /index.php/mainc/download HTTP/1.1" 200 656 "-" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-GB; rv:1.9.2.3) Gecko/20100401 Firefox/3.6.3" " does it say anything?
and I also tried to validate that page (the source i need) and got the answer: "500 Internal Server Error". Maybe there are other tools that can download source code? Maybe some can emulate the browser? because I can get the source code via browser.
fredass
Forum Newbie
Posts: 12
Joined: Tue Apr 06, 2010 4:18 pm

Re: php cURL problem

Post by fredass »

as i understand its possible to simulate browser with cURL, but cant find any example that works with my URL.. maybe because its HTTPS?
fredass
Forum Newbie
Posts: 12
Joined: Tue Apr 06, 2010 4:18 pm

Re: php cURL problem

Post by fredass »

I've solved the problem. That pages was blocking my connections from server. Had to use cURL to emulate web browser.

Code: Select all

$ch = curl_init();
	$url = "https://xxxxxxxx.xx/app/notice/notices.asp";
	$fp = fopen("./system/application/views/downloaded_html/home.html", "w") or die("file for source output was not opened");
	
	$headers = array 
	(
		'POST /app/notice/notices.asp HTTP/1.1',
		'Host:xxxxxxx.xx',  //your host
		'Connection: close',
		'Accept-Charset: ISO-8859-1,UTF-8;q=0.7,*;q=0.7',
		'Cache-Control: no',
		'Accept-Language: de,en;q=0.7,en-us;q=0.3',
		'Referer: http://web-sniffer.net/',
		'Content-type: application/x-www-form-urlencoded',
		'Content-length: 0'
	);

	curl_setopt($ch, CURLOPT_FILE, $fp);
	curl_setopt($ch, CURLOPT_URL, $url);
	curl_setopt($ch, CURLOPT_POST,1);	
	curl_setopt($ch, CURLOPT_HEADER, 0);
	curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);		
	curl_exec($ch);
	print_r(curl_getinfo($ch));
	curl_close($ch);
	fclose($fp);
	
	echo br(3);
	echo "done";
Post Reply