curl doesnt quite cut it

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
chopficaro
Forum Commoner
Posts: 68
Joined: Fri Jan 01, 2010 12:56 am

curl doesnt quite cut it

Post by chopficaro »

i am using curl to grab search results from web pages
i want to use it on this site
http://www.twitch.tv/search?query=dan+d ... dinh&stp=1
but the source code returned does not have the title, description etc in the source code
instead they have variables which look like {{title}}, {{description}} in its place
i have searched all of the java script files and all the other files in the source code for these variable values and cannot find them
but if the web browser can translate these variables into their values, there must be a way i can do it in php
does anyone know if there are other functions like curl that have the functionality i am looking for?
User avatar
Christopher
Site Administrator
Posts: 13596
Joined: Wed Aug 25, 2004 7:54 pm
Location: New York, NY, US

Re: curl doesnt quite cut it

Post by Christopher »

You will need to have a browser render the page after you fetch it. cURL just sends/gets requests -- it does not know how to render different data formats.
(#10850)
chopficaro
Forum Commoner
Posts: 68
Joined: Fri Jan 01, 2010 12:56 am

Re: curl doesnt quite cut it

Post by chopficaro »

is there a way to simulate browser rendering?
User avatar
tr0gd0rr
Forum Contributor
Posts: 305
Joined: Thu May 11, 2006 8:58 pm
Location: Utah, USA

Re: curl doesnt quite cut it

Post by tr0gd0rr »

chopficaro wrote:is there a way to simulate browser rendering?
CasperJS is an awesome way to use a virtual browser to open a site, manipulate dom, take screenshots, click elements, submit forms, etc. On Ubuntu, you can install it with `sudo apt-get install casperjs` or OSX `brew install casperjs`. It uses a full V8 browser engine and executes JavaScript like a normal browser.
Post Reply