screen scraping a site which uses AJAX

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
TheStoryTeller
Forum Newbie
Posts: 15
Joined: Fri May 01, 2009 10:13 am

screen scraping a site which uses AJAX

Post by TheStoryTeller »

Hi there, I'm new here so I'm not sure if this should be posted here.

I'm building a website which among other things, presents specific sports scores and statistics, taken from some other site.
I am forced to use screen scraping, which is fine for most sites, but one of them is giving me trouble.
The data in this site is presented in AJAX technology, which means the data of the page is changing, but the url is the same.
I need all the data. The html isn't populated with all the nesseary data, but I don't know how to access the entire data.
I did notice a javascript call that is activated in the code when the user is accessing manually the data.

how can I simulate the javascript call in php? is there any other way to access the data - which is available to all - by code?
User avatar
John Cartwright
Site Admin
Posts: 11470
Joined: Tue Dec 23, 2003 2:10 am
Location: Toronto
Contact:

Re: screen scraping a site which uses AJAX

Post by John Cartwright »

Take a look at a sniffer, such as Fiddler2 or better yet Firebug, which can intersept all http calls between the client and server. Basically, you'd just need to re-create the HTTP calls, especially since javascript only is a medium for XmlHttpRequest anywho.
TheStoryTeller
Forum Newbie
Posts: 15
Joined: Fri May 01, 2009 10:13 am

Re: screen scraping a site which uses AJAX

Post by TheStoryTeller »

McInfo wrote:I use Firebug (Firefox extension) to intercept AJAX requests on tvguide.com. There is a bit of manual labor because I have to copy-and-paste the response onto a form page on my localhost, then step through a series of processing scripts to get the data into my database.
By manual labor, do you mean actuall pressing ctrl-c followed by ctrl-v, or are you referring to a cumbersome code as manual labor... I'm sorry if I sound noob.. but if you have to do something by hand.. then what's the point of all this?
TheStoryTeller
Forum Newbie
Posts: 15
Joined: Fri May 01, 2009 10:13 am

Re: screen scraping a site which uses AJAX

Post by TheStoryTeller »

Thank You both for your responses. I'v been searching for an answer for several weeks now. Firebug was just what I needed!
Post Reply