PHP Developers Network

Using a cue system to deal with loop processes.
Page 1 of 1

Author:  bowlesj [ Thu Jan 15, 2015 6:45 am ]
Post subject:  Using a cue system to deal with loop processes.

Hi guys,

this is a followup idea for processing heavy loads. I agree about what was said about load balancing in my recent post and I am just itching to read about it to see how it distributes multiple tables over multiple disk drives. However I am currently working on the help for my website and I have had it in the back of my head that the loop processes of my website will need to have a cue system so users go through it (the loop processes) one at a time and they are prepared for it such that they can do a bit of other work until the computer beeps when it is done. So I woke up last night with the rough cue system code in my head. Below is the cue table and the pseudo-code I roughed in. Maybe someone could use this idea or maybe someone has tried this and has some thoughts on it. I am thinking it might be needed even if load balancing is in place some day.

Thanks John.

Syntax: [ Download ] [ Hide ]
The Cue Table:
        ProcessNumber (I have 3 processes that may need cueing for first come first serve processing)
        CueDateTime (write record out up front)
        ProcessDateTimeStart (updated once cue is ended and when process starts)
        ProcessDateTimeEnd (updated once when process ends)
Processing pseudo-code
        Give the user a popup telling them they may be cued and the best times to avoid cueing.
        Force the writing of times to a consistent eastern standard time (not sure how to do that).
        First thing upon user call to process write a cue record with the process # and CueDateTime
        TopOfLoopStart: (comes back here after the sleep command).
        Query Cue file on
                      ProcessNumber &
                      CueDateTime < myCueDateTime &
                      ProcessDateTime = null
                     (in other words check cue for anyone doing my process and anyone ahead of me).
        If # Cue Records ahead of me > 0 then
              Echo a message to user with # in cue before them & estimated time to start process.
              Sleep(Number of cue records before the user X estimated time to process);
              Maybe use USleep instead depending on process time estimates.
              After sleep is complete go back to TopOfLoopStart;
        Capture the ProcessDateTimeStart in memory
        Complete the process:
               First it zeros about 100 selected records on one of the two tables it is about to process.
               Next it matches 2 files/tables and adds to a counter on one of the two files upon match.
               One of the processes does one more pass in a different sequence to do some grouping.
        Capture the ProcessDateTimeEnd
        Write out the ProcessDateTimeStart and ProcessDateTimeEnd on the Cue record.
        Overlay the echo statements the user was seeing with a proper page and give a beep.
         On a weekly basis run a process that studies the Cue file process times.
         This process will be used to decide when to clear the cue table.
         It will also be used to tell the users when the least busy times are to avoid cuing.
         One thing I have not figured out is the time zone processing. Lets think about it.
                  All times would need to be consistent on the cue file (eastern standard time).
                  Figure out the best times for the user in Eastern standard time.
                  On the cities file have the time zones.
                  The user sees a page that uses their time zone to figure out the best times to run for their time zone
                  (it is converted from the eastern standard time on the file to the user's time zone).
        The Clearing Process:
               The clearing process should be cued.
               However when it checks the cue it does not test for dateTime less than the current time.
               Instead it tests for anything in the cue at all.
               When the cue is fully empty then the clear runs and it removes all records from the file.
               The structure of the file is kept so the next user can write a cue record out.

Author:  Celauran [ Thu Jan 15, 2015 7:53 am ]
Post subject:  Re: Using a cue system to deal with loop processes.

What kind of work are we talking about that you need a queuing system right off the bat? That sounds highly unusual and raises some flags for me. You could take a look at beanstalkd or IronMQ or some such, but first I'd take a good look at your code and try to figure out if you really need it.

Author:  bowlesj [ Thu Jan 15, 2015 11:45 am ]
Post subject:  Re: Using a cue system to deal with loop processes.

Thanks for responding Celauran. The system now (under local host) processes/updates one set of selected test records (about 50 I would say rather than 100) and it presents the page about as fast as I can snap my finger. It does it perfectly to match the objective of the website (this includes many test runs with netbeans). Basically the members will not complain if the pages come up as fast as I get them and if they are happy I will be happy. Roughly every week 5 users on average would need to run the same process on the same set of selected records (50 to 100 or maybe a bit more records). Right now I only allow one user to process this selected set of records at a time. Practically this means that maybe once a month one of them would get a message to try again a few seconds later (3 snaps of a finger lets say). Obviously asynchronous will not work here since they are going after the exact same set of selected records running a totaling process against them. Okay but other users will be processing(updating with this totaling process) other sets of selected records on the same table at the same time . I have no restrictions at the present time for other users going after a different set of selected records. I am assuming that asynchronous may be of value in this case such that maybe beanstalkd or IronMQ is the answer. I don't know. If the need is there and they do not cost much and I do some backups I guess I could give that a try to see what happens. After I posted, I was thinking about the amount of potential wait time of the cue system described above. It could be several years before the website could be popular enough to create a wait time of 1 minute at any one time. It does not take long to snap one's finger 10 times. However I guess the web is a lot slower than localhost and I have a very fast test machine. I guess I will know the difference right away when I do the live tests come March/April some time. The market of this website is a small fraction the size of facebook (probably 1/1000 the size or even less). Its a nitch market. However in theory it could get 1000 members per city in a big city. Time will tell.

Author:  Celauran [ Thu Jan 15, 2015 12:01 pm ]
Post subject:  Re: Using a cue system to deal with loop processes.

1/1000 of Facebook's users is still 600,000 people.

Author:  bowlesj [ Thu Jan 15, 2015 12:07 pm ]
Post subject:  Re: Using a cue system to deal with loop processes.

Yeah, true - LOL. I am assuming that is all of Facebook and yes there are a lot of cities that may join my website (very few in the boonies would join). 600,000 / 5 = 120,000 going after the same table to update different sets of 100 records at the same time. Yep, that is why I am thinking about this - LOL.

Page 1 of 1 All times are UTC - 5 hours
Powered by phpBB® Forum Software © phpBB Group