Parsing n number of urls without timeout

PHP programming forum. Ask questions or help people concerning PHP code. Don't understand a function? Need help implementing a class? Don't understand a class? Here is where to ask. Remember to do your homework!

Moderator: General Moderators

Post Reply
kkonline
Forum Contributor
Posts: 251
Joined: Thu Aug 16, 2007 12:54 am

Parsing n number of urls without timeout

Post by kkonline »

Hi All,
I am working on a rss feed aggregator, the concept/algorithm is

1>I connect to a source url
2>Parse the text content
3>Wite on destination server in steps of 40KB
4>After every 40KB the content is trasferred to next file (myfile.2) and then on reaching threshhold 40KB to myfile3.txt

These steps will run for each url

Now the issue i am facing in this is after I connect an grab the data from source1 and then if i grab the data from source 2 get get script timeout.

Is there any possible workaround/logic.trick so that i can connect to any number of urls and then grab the data from all at once without this timeout.

Increasing the time to process a script is the solution i am not looking for... anyother idea?
maneetpuri
Forum Commoner
Posts: 60
Joined: Tue Oct 07, 2008 6:32 am

Re: Parsing n number of urls without timeout

Post by maneetpuri »

Hi,

Increasing the script timeout time is definitely a solution but might not work if data to be fetched is more, what you can do is break the task of fetching the data into part. Once the parsing has started keep loging the amount of data parsed and when reach a particualr point flag it, stop the execution and restart the execution but now from the point where it stopped.

I hope it sounds logic to you.
Last edited by onion2k on Sat Oct 11, 2008 6:05 am, edited 2 times in total.
Reason: Additional, unnecessary link that should be in a sig rather than the post itself removed.
handride
Forum Newbie
Posts: 1
Joined: Tue Jun 09, 2009 1:23 pm

Re: Parsing n number of urls without timeout

Post by handride »

Hi I'm glad this isn't a new problem, and I think my question is a bit more complex, but it does come down to RSS aggregation in PHP.

I have over 100 RSS's I'd like to follow, and might have to add more

RSS 1-25 is group A, 26-50 is group B, and 51-60 is C, 61-100 is D (for argument's sake)

when a person comes to the page all 100 feeds are shown sorted by published date, however the visitor has the ability to click on group A and just get those feeds.

I'm currently using magpie, but i'm just a newb and really not a real programmer, i've hired people to help me with my site, and I just want to make sure I'm paying for nothing.

the site is here: http://tbablogs.com/index.php (it obviously crashes a lot) I'm just looking on how to implement the above if possible? any type of help appreciated.
Post Reply