Page 1 of 1

curl doesnt quite cut it

Posted: Wed Aug 22, 2012 8:59 pm
by chopficaro
i am using curl to grab search results from web pages
i want to use it on this site
http://www.twitch.tv/search?query=dan+d ... dinh&stp=1
but the source code returned does not have the title, description etc in the source code
instead they have variables which look like {{title}}, {{description}} in its place
i have searched all of the java script files and all the other files in the source code for these variable values and cannot find them
but if the web browser can translate these variables into their values, there must be a way i can do it in php
does anyone know if there are other functions like curl that have the functionality i am looking for?

Re: curl doesnt quite cut it

Posted: Thu Aug 23, 2012 10:22 am
by Christopher
You will need to have a browser render the page after you fetch it. cURL just sends/gets requests -- it does not know how to render different data formats.

Re: curl doesnt quite cut it

Posted: Thu Aug 23, 2012 1:26 pm
by chopficaro
is there a way to simulate browser rendering?

Re: curl doesnt quite cut it

Posted: Fri Aug 24, 2012 1:54 pm
by tr0gd0rr
chopficaro wrote:is there a way to simulate browser rendering?
CasperJS is an awesome way to use a virtual browser to open a site, manipulate dom, take screenshots, click elements, submit forms, etc. On Ubuntu, you can install it with `sudo apt-get install casperjs` or OSX `brew install casperjs`. It uses a full V8 browser engine and executes JavaScript like a normal browser.