Need help on scrapping script

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
php_coder
Forum Newbie
Posts: 3
Joined: Mon Aug 24, 2009 9:47 pm

Need help on scrapping script

Post by php_coder »

<?php

session_start();

$_SESSION['number']=$_POST["number"];

$userAgent = "Googlebot/2.1 (+http://www.google.com/bot.html)";

$target_url="http://example.com/".$_SESSION['number'];

$ch = curl_init();
curl_setopt($ch, CURLOPT_URL,$target_url);
curl_setopt($ch, CURLOPT_USERAGENT, $userAgent);
curl_setopt($ch, CURLOPT_FAILONERROR, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_COOKIEFILE, "cookiefile");
curl_setopt($ch, CURLOPT_COOKIEJAR, "cookiefile"); # SAME cookiefile
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_RETURNTRANSFER,true);
curl_setopt($ch, CURLOPT_TIMEOUT, 0);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

$html= curl_exec($ch);
echo $html;
$dom = new DOMDocument();
@$dom->loadHTML($html);

$all_div=$dom->getElementsByTagName('div');

foreach($all_div as $div)
{
if($div->getAttribute('class')=="d1-2")
{
echo "<center>";
echo "Example doesn't exist";
echo "</center>";
}
else
if($div->getAttribute('class')=="uinf-2-2-4-2")
{
echo "<center>";
echo "Example exist";
echo "</center>";
}


}


?>

This script runs perfectly on the localhost, but when i run the script on the server(yahoo small business). It just echo's html page, it doesn't output example doesn't exist or example exist.

I think script time's out, not sure what the problem is. Any suggestions?
User avatar
Ambush Commander
DevNet Master
Posts: 3698
Joined: Mon Oct 25, 2004 9:29 pm
Location: New Jersey, US

Re: Need help on scrapping script

Post by Ambush Commander »

You can find out if it's a timeout by adding the following lines to your code:

Code: Select all

error_reporting(E_ALL);
ini_set('display_errors', 1);
php_coder
Forum Newbie
Posts: 3
Joined: Mon Aug 24, 2009 9:47 pm

Re: Need help on scrapping script

Post by php_coder »

Ambush Commander wrote:You can find out if it's a timeout by adding the following lines to your code:

Code: Select all

error_reporting(E_ALL);
ini_set('display_errors', 1);

Thanks for the code, it was not exactly a time out problem. Problem was with the yahoo server, they only support php 4, it worked perfectly on php 5. I think php 4 doesn't support dom.
User avatar
Ambush Commander
DevNet Master
Posts: 3698
Joined: Mon Oct 25, 2004 9:29 pm
Location: New Jersey, US

Re: Need help on scrapping script

Post by Ambush Commander »

php_coder
Forum Newbie
Posts: 3
Joined: Mon Aug 24, 2009 9:47 pm

Re: Need help on scrapping script

Post by php_coder »

[quote="Ambush Commander"]Lies.

I never said, that i am sure about php 4 not supporting dom. I just said i think.
User avatar
Ambush Commander
DevNet Master
Posts: 3698
Joined: Mon Oct 25, 2004 9:29 pm
Location: New Jersey, US

Re: Need help on scrapping script

Post by Ambush Commander »

It was meant in a humorous manner ;-) domxml does not actually work that well, and has given me loads of grief in the past.
Post Reply