cURL Experiments

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
UniqueIdeaMan
Forum Contributor
Posts: 197
Joined: Wed Jan 18, 2017 3:43 pm

cURL Experiments

Post by UniqueIdeaMan »

Ladies & Gentlemen!

Oops! Let me try again:
Gentle Ladies & Hard Men (after-all, it's the ladies who are gentle compared to men and men hard, rough & tough compared to the ladies)!
And no, don't jest by saying "hard men" sounds like men having an erec**ion as some foolish men joked on other forums (busines/internet marketing) few months (or was it a yr back ?) back! :lol:
Get used to me addressing you as "hard men" because that's how I'm gonna do it frequently. :rofl:

Anyway, this thread is about cURL.
I will try building a unique script based on cURL. But, let's get rolling to learn first!
I have some viral traffic & viral money earning ideas and cURL will impliment them. Stick around and see how deep the rabbit hole is and what comes out of it! (Not joking!).
Last edited by UniqueIdeaMan on Tue May 23, 2017 6:25 pm, edited 1 time in total.
UniqueIdeaMan
Forum Contributor
Posts: 197
Joined: Wed Jan 18, 2017 3:43 pm

Re: cURL Experiments

Post by UniqueIdeaMan »

cURL Sample 1:


Why do you reckon the following code sample is not showing any page loading ? I see a complete blank page loading on my xampp.

Code: Select all

<?php

//This code was found on: http://www.binarytides.com/php-curl-tutorial-beginners/
//gets the data from a URL
function get_url($url) 
{
    $ch = curl_init();
     
    if($ch === false)
    {
        die('Failed to create curl object');
    }
     
    $timeout = 5;
    curl_setopt($ch, CURLOPT_URL, $url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
    $data = curl_exec($ch);
    curl_close($ch);
    return $data;
}
 
echo get_url('http://www.apple.com/');
?>

UniqueIdeaMan
Forum Contributor
Posts: 197
Joined: Wed Jan 18, 2017 3:43 pm

Re: cURL Experiments

Post by UniqueIdeaMan »

cURL Sample 2:

Is it true that the following short code is just as good as the one mentioned on my previous post ?

Code: Select all


<?php 

//This code was found on: http://www.binarytides.com/php-curl-tutorial-beginners/
//2nd Example
//The above GET request to a url can be done in a much simpler way like this:

//Make a HTTP GET request and print it (requires allow_url_fopen to be enabled)
echo file_get_contents('http://www.apple.com/');

?>

Last edited by UniqueIdeaMan on Tue May 23, 2017 5:56 pm, edited 1 time in total.
UniqueIdeaMan
Forum Contributor
Posts: 197
Joined: Wed Jan 18, 2017 3:43 pm

Re: cURL Experiments

Post by UniqueIdeaMan »

cURL Sample 3:

Can you figure-out or atleast guess why the following code is better than the previous 2 ?
What benefits do you see in it than the other 2 code samples ?
No, I'm not testing you but trying to learn from you.

Code: Select all


<?php 

//3rd Option
//This code was found on: http://www.binarytides.com/php-curl-tutorial-beginners/
//Calling the curl_setopt function again and again to set the options is a bit tedious. There is a useful function called curl_setopt_array that takes an array of options and sets them all at once. Here is a quick example:
//Make a HTTP GET request and print it (requires allow_url_fopen to be enabled)
echo file_get_contents('http://www.apple.com/');
curl_setopt_array($ch, array(
    CURLOPT_URL => $url ,
    CURLOPT_RETURNTRANSFER => 1,
    CURLOPT_CONNECTTIMEOUT => $timeout ,
));

?>

And, what is meant by the following:
"Calling the curl_setopt function again and again to set the options is a bit tedious. There is a useful function called curl_setopt_array that takes an array of options and sets them all at once.".

Can you elaborate it ? Because, if I understand it then I'll understand the benefits of this code over the previous 2.

Thanks!
UniqueIdeaMan
Forum Contributor
Posts: 197
Joined: Wed Jan 18, 2017 3:43 pm

Re: cURL Experiments

Post by UniqueIdeaMan »

Guys,

On this cURL tutorial:
http://www.binarytides.com/php-curl-tutorial-beginners/

Under the section **"Make GET requests - fetch a url"**, you will see 3 blocks of code.
I'm referring to the 3rd one that looks like this:

Code: Select all


    curl_setopt_array($ch, array(
        CURLOPT_URL => $url ,
        CURLOPT_RETURNTRANSFER => 1,
        CURLOPT_CONNECTTIMEOUT => $timeout ,
    ));



How come no url is mentioned in that code ?
I can now see why my code sample 3 is not working. See my 4th post above.
What do you think that code should look like ? Care to show an example, how you'd do things ?
UniqueIdeaMan
Forum Contributor
Posts: 197
Joined: Wed Jan 18, 2017 3:43 pm

Re: cURL Experiments

Post by UniqueIdeaMan »

Guys,

What I am trying to do is load google.com with cURL and no matter what link is on the page, be it a google link or another domain), I want the script to precede: http://mydomain.com/tracker.php?

So, if the page contains a link like this:

http://somedomain.com/

then that should be replaced with:

http://mydomain.com/tracker.php?http://somedomain.com/

And, if the page contains link like this:

http://subdomain.somedomain.com/dir/sub-dir/page.html

Then, I want it to be replaced to:

http://mydomain.com/tracker.php?http:// ... /page.html

You get my point ?

Here's my code but I get error:

Warning: preg_replace(): Delimiter must not be alphanumeric or backslash in C:\xampp\htdocs\test\CURL_experiment1.php on line 13

Here's my code:

Code: Select all

<?php
$url = "http://google.com";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "$url");
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_HEADER, 5);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$result = curl_exec($ch);
curl_close($ch);

$pattern = 'http://';
$replacement = 'http://mydomain.com/tracker.php?';
echo preg_replace($pattern, $replacement, $result);

?>
UniqueIdeaMan
Forum Contributor
Posts: 197
Joined: Wed Jan 18, 2017 3:43 pm

Re: cURL Experiments

Post by UniqueIdeaMan »

What does the following header mean (ebay.com):

**HTTP/1.1 200 OK X-EBAY-C-REQUEST-ID: ri=N79YSYGvO9se,rci=qZJNjVUUeUF9xCaS RlogId: t6e%60cckjkb9%3Fvo%7Bccbgmijf%28vo%7B%287570577-15c59d3449c-0x129 X-Frame-Options: SAMEORIGIN X-Frame-Options: SAMEORIGIN X-Content-Type-Options: nosniff X-XSS-Protection: 1; mode=block Cache-Control: private Pragma: no-cache Content-Type: text/html;charset=utf-8 Content-Language: en-US Server: ebay server X-EdgeConnect-MidMile-RTT: 172 X-EdgeConnect-Origin-MEX-Latency: 261 X-EdgeConnect-Cache-Status: 0 Date: Tue, 30 May 2017 14:47:17 GMT Transfer-Encoding: chunked Connection: keep-alive Connection: Transfer-Encoding Set-Cookie: JSESSIONID=B9699C4784AF48E8C158816A1094F78B; Path=/; HttpOnly Set-Cookie: ebay=%5Esbf%3D%23%5E;Domain=.ebay.com;Path=/ Set-Cookie: dp1=bu1p/QEBfX0BAX19AQA**5b0eb975^bl/BD5cefecf5^;Domain=.ebay.com;Expires=Thu, 30-May-2019 14:47:17 GMT;Path=/ Set-Cookie: s=CgAD4ACBZLtd1NTlkMzQ0OGMxNWMwYTg4OGQ4YzYzYTYzZmZmOGJhNjhjlJyS;Domain=.ebay.com;Path=/; HttpOnly Set-Cookie: nonsession=CgADLAAFZLYz9MQDKACBik4d1NTlkMzQ0OGMxNWMwYTg4OGQ4YzYzYTYzZmZmOGJhNjgpg9Wr;Domain=.ebay.com;Expires=Wed, 30-May-2018 14:47:17 GMT;Path=/**


I have a feeling the header states that the visitor is the same visitor and so is this the case why I'm unable to precede the following on all links present on ebay homepage ?
(NOTE: I'm getting cURL to fetch ebay homepage).

http://mydomain.com/tracker.php?

Check the following code. Run it on your xamp/wamp and then hover your mouse over the links present on ebay's homepage. Do you see all links starting with:

http://mydomain.com/tracker.php?

Yes, or no ? i used to see it. And so a YES from here from lastnight.
But tonight, the answer is a NO.
Now, I'm confused why is that ? Anything wrong with my coding ?

Code: Select all

<?php
$url = "http://www.ebay.com";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "$url");
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_HEADER, 5);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$result = curl_exec($ch);
curl_close($ch);

/* pattern can be either 2:
$pattern = '#http://#';
$pattern = '/http:\/\//';
*/

$pattern = '/http:\/\//';
$replacement = 'http://mydomain.com/tracker.php?';

$pattern = '/https:\/\//';
$replacement = 'http://mydomain.com/tracker.php?';

$pattern = '/localhost\//';
$replacement = 'http://mydomain.com/tracker.php?';

echo str_replace($pattern, $replacement, $result);

?>

What I'm doing here is just replacing the "http://" with:

http://mydomain.com/tracker.php?

But instead of using the replace function, is there a better way to precede the following on all links found on ebay homepage ? (The technique should work on any homepage and not be dependant on ebay homepage only).
UniqueIdeaMan
Forum Contributor
Posts: 197
Joined: Wed Jan 18, 2017 3:43 pm

Re: cURL Experiments

Post by UniqueIdeaMan »

Folks,

I'm afraid the following preg_replace is not doing the job. Can you think of a better one ?
Actually, if you can come-up with one that does the following then I'd appreciate it:

1. Replace 'https://' with: 'http://mydomain.com'.
2. Replace 'http://' with: 'http://mydomain.com'.
3. Replace 'www.' with: 'http://mydomain.com'.
4. Replace all subdomains and sub sub domains etc. with: 'http://mydomain.com'.
eg1. Replace 'mail.domain.com' with 'http://mydomain.com'.
eg2. Replace 'ny.mail.domain.com' with 'http://mydomain.com'.
eg3. Replace 'europe.spain.mail.domain.com' with 'http://mydomain.com'.
eg4. Replace 'west.europe.deutchland.mail.domain.com' with 'http://mydomain.com'.
And so on. You get the picture.

I tried the following code but it's not working. Instead of replacing things with 'http://mydomain.com' it is replacing with: '%24url//'. And that is a big No! No!

Code: Select all


<?php
$url = "http://google.com";
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, "$url");
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, TRUE);
curl_setopt($ch, CURLOPT_HEADER, 5);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, TRUE);
$result = curl_exec($ch);
curl_close($ch);
$result = preg_replace("#(<\s*a\s+[^>]*href\s*=\s*[\"'])(?!http)([^\"'>]+)([\"'>]+)#",'$1http://$url/$2$3', $result);
echo $result
?>

Cheers!
User avatar
Pazuzu156
Forum Contributor
Posts: 241
Joined: Sat Nov 20, 2010 9:00 pm
Location: GA, USA
Contact:

Re: cURL Experiments

Post by Pazuzu156 »

preg_replace does just fine. Though I tend to use preg_replace_callback instead, so I can use an anonymous function to handle the replacements
- Kaleb Klein
------------------------------------
Web Developer | Software Developer
https://kalebklein.com
PGP Key: https://keybase.io/pazuzu156
UniqueIdeaMan
Forum Contributor
Posts: 197
Joined: Wed Jan 18, 2017 3:43 pm

Re: cURL Experiments

Post by UniqueIdeaMan »

Pazuzu156 wrote:preg_replace does just fine. Though I tend to use preg_replace_callback instead, so I can use an anonymous function to handle the replacements
How about showing an example ? Take my code, for starters, and build your code sample from there.

Thanks!
Post Reply