Hi there, I'm new here so I'm not sure if this should be posted here.
I'm building a website which among other things, presents specific sports scores and statistics, taken from some other site.
I am forced to use screen scraping, which is fine for most sites, but one of them is giving me trouble.
The data in this site is presented in AJAX technology, which means the data of the page is changing, but the url is the same.
I need all the data. The html isn't populated with all the nesseary data, but I don't know how to access the entire data.
I did notice a javascript call that is activated in the code when the user is accessing manually the data.
how can I simulate the javascript call in php? is there any other way to access the data - which is available to all - by code?
screen scraping a site which uses AJAX
Moderator: General Moderators
-
TheStoryTeller
- Forum Newbie
- Posts: 15
- Joined: Fri May 01, 2009 10:13 am
- John Cartwright
- Site Admin
- Posts: 11470
- Joined: Tue Dec 23, 2003 2:10 am
- Location: Toronto
- Contact:
Re: screen scraping a site which uses AJAX
Take a look at a sniffer, such as Fiddler2 or better yet Firebug, which can intersept all http calls between the client and server. Basically, you'd just need to re-create the HTTP calls, especially since javascript only is a medium for XmlHttpRequest anywho.
-
TheStoryTeller
- Forum Newbie
- Posts: 15
- Joined: Fri May 01, 2009 10:13 am
Re: screen scraping a site which uses AJAX
By manual labor, do you mean actuall pressing ctrl-c followed by ctrl-v, or are you referring to a cumbersome code as manual labor... I'm sorry if I sound noob.. but if you have to do something by hand.. then what's the point of all this?McInfo wrote:I use Firebug (Firefox extension) to intercept AJAX requests on tvguide.com. There is a bit of manual labor because I have to copy-and-paste the response onto a form page on my localhost, then step through a series of processing scripts to get the data into my database.
-
TheStoryTeller
- Forum Newbie
- Posts: 15
- Joined: Fri May 01, 2009 10:13 am
Re: screen scraping a site which uses AJAX
Thank You both for your responses. I'v been searching for an answer for several weeks now. Firebug was just what I needed!