Page 1 of 1

curl and javascript

Posted: Tue May 03, 2011 2:55 am
by JakeJ
I am trying to figure out how to use curl to download all the content that would be loaded on to a web browser that has javascript enabled.

Background: Using Fiddler, I load msn.com for example. Javascript is enabled in my browser so I can get all of the content behind the tabs that way (it loads in the background). When I use my curl script this obviously doesn't work since there's no javascript functionality that allows me to load that additional content.

How do I go about loading the additional content or at least simulating the tab mouseovers (or clicks) necessary to load the content? There are no forms involved in what I need to do.

Any and all help on this matter is appreciated.

**Edit**
I should further clarify that what I am really after is to have all the content loaded in to a string so I do some regex parsing for particular strings of text. I'm not looking to simply display the contents as they come across.

Re: curl and javascript

Posted: Fri May 06, 2011 1:55 pm
by tr0gd0rr
You'll have to look at the ajax requests and use curl to call the same URLs. You may need to have curl make an initial request to the page to get a session cookie and then send that session cookie to the ajax URL. This kind of script gets very complicated very quickly and it will break if the web site changes the URLs or methodology of their ajax calls. You may want to think about other approaches to get the data you need.