Scan and auto-download?
Posted: Tue Jan 05, 2010 6:01 pm
I have a few questions in regards to feasibility of a few ideas I have. My current project involves mainly, data collection and submission from various sites. Given that almost all sites rely on RSS, I've been able to branch some of them and archive the basics.
Where my question comes in is that while we can manually go through websites to download and assign their media and content to our database, as they produce and submit more information to the RSS, is there a way to take advantage of that?
More specifically, if I have an RSS feed that gives me the link to the page I need with the correct id for the download, is there some way through PHP to scan the source code for certain keywords to use in the database submission, as well as download with that unique id?
If anyone can point me in the direction of the functions I would need if this is indeed possible, or any other suggestions, that would be greatly appreciated.
Where my question comes in is that while we can manually go through websites to download and assign their media and content to our database, as they produce and submit more information to the RSS, is there a way to take advantage of that?
More specifically, if I have an RSS feed that gives me the link to the page I need with the correct id for the download, is there some way through PHP to scan the source code for certain keywords to use in the database submission, as well as download with that unique id?
If anyone can point me in the direction of the functions I would need if this is indeed possible, or any other suggestions, that would be greatly appreciated.