Writing a queuing system.

Not for 'how-to' coding questions but PHP theory instead, this forum is here for those of us who wish to learn about design aspects of programming with PHP.

Moderator: General Moderators

User avatar
Christopher
Site Administrator
Posts: 13596
Joined: Wed Aug 25, 2004 7:54 pm
Location: New York, NY, US

Post by Christopher »

jshpro2 wrote:That is the documentation on how to use table locking, nowhere does it say INSERTS / UPDATES lock the table, correct me if I am wrong
"To achieve a very high lock speed, MySQL uses table locking (instead of page, row, or column locking) for all storage engines except InnoDB and BDB." -- The default table type MyISAM uses table locking.

"Table locking enables many threads to read from a table at the same time, but if a thread wants to write to a table, it must first get exclusive access. During the update, all other threads that want to access this particular table must wait until the update is done."
(#10850)
User avatar
onion2k
Jedi Mod
Posts: 5263
Joined: Tue Dec 21, 2004 5:03 pm
Location: usrlab.com

Post by onion2k »

If I were to write a system that only read a table and notified the user when their job was completed I may as well just email them a link to the finished file .. no need for a queue at all .. just a script that works through all the available jobs.

The point of having a queue is that I can avoid doing any processing for users who leave. I don't think I can avoid writing to a table every time someone pings the queue.
User avatar
Christopher
Site Administrator
Posts: 13596
Joined: Wed Aug 25, 2004 7:54 pm
Location: New York, NY, US

Post by Christopher »

onion2k wrote:The point of having a queue is that I can avoid doing any processing for users who leave.
I don't see how either method avoids having to clean out abandoned records on a regular basis. If you depend on the user updating a timestamp and they stop updating then something has to get rid of that record based on exceeding some timeframe.
onion2k wrote:I don't think I can avoid writing to a table every time someone pings the queue.
What I proposed only has three writes: when the user adds the entry to the queue, when the server is done processing the queue entry, and when the user completes the download. In between the server and user are polling with reads only.
(#10850)
User avatar
onion2k
Jedi Mod
Posts: 5263
Joined: Tue Dec 21, 2004 5:03 pm
Location: usrlab.com

Post by onion2k »

arborint wrote:
onion2k wrote:The point of having a queue is that I can avoid doing any processing for users who leave.
I don't see how either method avoids having to clean out abandoned records on a regular basis. If you depend on the user updating a timestamp and they stop updating then something has to get rid of that record based on exceeding some timeframe.
It doesn't avoid cleaning out abandoned records. It avoids running the script that the queue is controlling for users that haven't stuck around. Cleaning the records will take, maybe, 1/10th of a second. The script the users will be waiting for takes anything up to a minute to run. I think thats a fair trade off.
arborint wrote:
onion2k wrote:I don't think I can avoid writing to a table every time someone pings the queue.
What I proposed only has three writes: when the user adds the entry to the queue, when the server is done processing the queue entry, and when the user completes the download. In between the server and user are polling with reads only.
But what you propose doesn't allow me to abandon jobs in the queue if the user is no longer waiting for their file. That's what I need.
User avatar
BDKR
DevNet Resident
Posts: 1207
Joined: Sat Jun 08, 2002 1:24 pm
Location: Florida
Contact:

Post by BDKR »

Honestly you guys, I don't understand why you'd want to use a Database for such an operation. This seems kind a Golden Hammerish.

Apache uses something called a Scoreboard wich is a shared memory construct. Here is a description taken from activestate.com talking about the Perl Apache::Scoreboard module...
DESCRIPTION

Apache keeps track of server activity in a structure known as the scoreboard. There is a slot in the scoreboard for each child server, containing information such as status, access count, bytes served and cpu time. This same information is used by mod_status to provide current server statistics in a human readable form.
Once again, this is an shared memory construct. Remember that Apache 1.3x uses forked child processes which means each of these processes are trying to access this area of memory independently of one another or some other orchestrating mechanism.

Now PHP supports doing this kind of thing and this algorithm would be a good deal faster then using MySQL for something it wasn't really intended.
Post Reply