Using a cue system to deal with loop processes.
Posted: Thu Jan 15, 2015 5:45 am
Hi guys,
this is a followup idea for processing heavy loads. I agree about what was said about load balancing in my recent post and I am just itching to read about it to see how it distributes multiple tables over multiple disk drives. However I am currently working on the help for my website and I have had it in the back of my head that the loop processes of my website will need to have a cue system so users go through it (the loop processes) one at a time and they are prepared for it such that they can do a bit of other work until the computer beeps when it is done. So I woke up last night with the rough cue system code in my head. Below is the cue table and the pseudo-code I roughed in. Maybe someone could use this idea or maybe someone has tried this and has some thoughts on it. I am thinking it might be needed even if load balancing is in place some day.
Thanks John.
this is a followup idea for processing heavy loads. I agree about what was said about load balancing in my recent post and I am just itching to read about it to see how it distributes multiple tables over multiple disk drives. However I am currently working on the help for my website and I have had it in the back of my head that the loop processes of my website will need to have a cue system so users go through it (the loop processes) one at a time and they are prepared for it such that they can do a bit of other work until the computer beeps when it is done. So I woke up last night with the rough cue system code in my head. Below is the cue table and the pseudo-code I roughed in. Maybe someone could use this idea or maybe someone has tried this and has some thoughts on it. I am thinking it might be needed even if load balancing is in place some day.
Thanks John.
Code: Select all
The Cue Table:
ProcessNumber (I have 3 processes that may need cueing for first come first serve processing)
CueDateTime (write record out up front)
ProcessDateTimeStart (updated once cue is ended and when process starts)
ProcessDateTimeEnd (updated once when process ends)
Processing pseudo-code
Give the user a popup telling them they may be cued and the best times to avoid cueing.
Force the writing of times to a consistent eastern standard time (not sure how to do that).
First thing upon user call to process write a cue record with the process # and CueDateTime
TopOfLoopStart: (comes back here after the sleep command).
Query Cue file on
ProcessNumber &
CueDateTime < myCueDateTime &
ProcessDateTime = null
(in other words check cue for anyone doing my process and anyone ahead of me).
If # Cue Records ahead of me > 0 then
Echo a message to user with # in cue before them & estimated time to start process.
Sleep(Number of cue records before the user X estimated time to process);
Maybe use USleep instead depending on process time estimates.
After sleep is complete go back to TopOfLoopStart;
Capture the ProcessDateTimeStart in memory
Complete the process:
First it zeros about 100 selected records on one of the two tables it is about to process.
Next it matches 2 files/tables and adds to a counter on one of the two files upon match.
One of the processes does one more pass in a different sequence to do some grouping.
Capture the ProcessDateTimeEnd
Write out the ProcessDateTimeStart and ProcessDateTimeEnd on the Cue record.
Overlay the echo statements the user was seeing with a proper page and give a beep.
Maintenance:
On a weekly basis run a process that studies the Cue file process times.
This process will be used to decide when to clear the cue table.
It will also be used to tell the users when the least busy times are to avoid cuing.
One thing I have not figured out is the time zone processing. Lets think about it.
All times would need to be consistent on the cue file (eastern standard time).
Figure out the best times for the user in Eastern standard time.
On the cities file have the time zones.
The user sees a page that uses their time zone to figure out the best times to run for their time zone
(it is converted from the eastern standard time on the file to the user's time zone).
The Clearing Process:
The clearing process should be cued.
However when it checks the cue it does not test for dateTime less than the current time.
Instead it tests for anything in the cue at all.
When the cue is fully empty then the clear runs and it removes all records from the file.
The structure of the file is kept so the next user can write a cue record out.