Hi All,
I am working on a rss feed aggregator, the concept/algorithm is
1>I connect to a source url
2>Parse the text content
3>Wite on destination server in steps of 40KB
4>After every 40KB the content is trasferred to next file (myfile.2) and then on reaching threshhold 40KB to myfile3.txt
These steps will run for each url
Now the issue i am facing in this is after I connect an grab the data from source1 and then if i grab the data from source 2 get get script timeout.
Is there any possible workaround/logic.trick so that i can connect to any number of urls and then grab the data from all at once without this timeout.
Increasing the time to process a script is the solution i am not looking for... anyother idea?
Parsing n number of urls without timeout
Moderator: General Moderators
-
maneetpuri
- Forum Commoner
- Posts: 60
- Joined: Tue Oct 07, 2008 6:32 am
Re: Parsing n number of urls without timeout
Hi,
Increasing the script timeout time is definitely a solution but might not work if data to be fetched is more, what you can do is break the task of fetching the data into part. Once the parsing has started keep loging the amount of data parsed and when reach a particualr point flag it, stop the execution and restart the execution but now from the point where it stopped.
I hope it sounds logic to you.
Increasing the script timeout time is definitely a solution but might not work if data to be fetched is more, what you can do is break the task of fetching the data into part. Once the parsing has started keep loging the amount of data parsed and when reach a particualr point flag it, stop the execution and restart the execution but now from the point where it stopped.
I hope it sounds logic to you.
Last edited by onion2k on Sat Oct 11, 2008 6:05 am, edited 2 times in total.
Reason: Additional, unnecessary link that should be in a sig rather than the post itself removed.
Reason: Additional, unnecessary link that should be in a sig rather than the post itself removed.
Re: Parsing n number of urls without timeout
Hi I'm glad this isn't a new problem, and I think my question is a bit more complex, but it does come down to RSS aggregation in PHP.
I have over 100 RSS's I'd like to follow, and might have to add more
RSS 1-25 is group A, 26-50 is group B, and 51-60 is C, 61-100 is D (for argument's sake)
when a person comes to the page all 100 feeds are shown sorted by published date, however the visitor has the ability to click on group A and just get those feeds.
I'm currently using magpie, but i'm just a newb and really not a real programmer, i've hired people to help me with my site, and I just want to make sure I'm paying for nothing.
the site is here: http://tbablogs.com/index.php (it obviously crashes a lot) I'm just looking on how to implement the above if possible? any type of help appreciated.
I have over 100 RSS's I'd like to follow, and might have to add more
RSS 1-25 is group A, 26-50 is group B, and 51-60 is C, 61-100 is D (for argument's sake)
when a person comes to the page all 100 feeds are shown sorted by published date, however the visitor has the ability to click on group A and just get those feeds.
I'm currently using magpie, but i'm just a newb and really not a real programmer, i've hired people to help me with my site, and I just want to make sure I'm paying for nothing.
the site is here: http://tbablogs.com/index.php (it obviously crashes a lot) I'm just looking on how to implement the above if possible? any type of help appreciated.