Page 2 of 2

Posted: Thu Feb 09, 2006 11:33 pm
by Christopher
jshpro2 wrote:That is the documentation on how to use table locking, nowhere does it say INSERTS / UPDATES lock the table, correct me if I am wrong
"To achieve a very high lock speed, MySQL uses table locking (instead of page, row, or column locking) for all storage engines except InnoDB and BDB." -- The default table type MyISAM uses table locking.

"Table locking enables many threads to read from a table at the same time, but if a thread wants to write to a table, it must first get exclusive access. During the update, all other threads that want to access this particular table must wait until the update is done."

Posted: Fri Feb 10, 2006 3:00 am
by onion2k
If I were to write a system that only read a table and notified the user when their job was completed I may as well just email them a link to the finished file .. no need for a queue at all .. just a script that works through all the available jobs.

The point of having a queue is that I can avoid doing any processing for users who leave. I don't think I can avoid writing to a table every time someone pings the queue.

Posted: Fri Feb 10, 2006 3:15 am
by Christopher
onion2k wrote:The point of having a queue is that I can avoid doing any processing for users who leave.
I don't see how either method avoids having to clean out abandoned records on a regular basis. If you depend on the user updating a timestamp and they stop updating then something has to get rid of that record based on exceeding some timeframe.
onion2k wrote:I don't think I can avoid writing to a table every time someone pings the queue.
What I proposed only has three writes: when the user adds the entry to the queue, when the server is done processing the queue entry, and when the user completes the download. In between the server and user are polling with reads only.

Posted: Fri Feb 10, 2006 4:30 am
by onion2k
arborint wrote:
onion2k wrote:The point of having a queue is that I can avoid doing any processing for users who leave.
I don't see how either method avoids having to clean out abandoned records on a regular basis. If you depend on the user updating a timestamp and they stop updating then something has to get rid of that record based on exceeding some timeframe.
It doesn't avoid cleaning out abandoned records. It avoids running the script that the queue is controlling for users that haven't stuck around. Cleaning the records will take, maybe, 1/10th of a second. The script the users will be waiting for takes anything up to a minute to run. I think thats a fair trade off.
arborint wrote:
onion2k wrote:I don't think I can avoid writing to a table every time someone pings the queue.
What I proposed only has three writes: when the user adds the entry to the queue, when the server is done processing the queue entry, and when the user completes the download. In between the server and user are polling with reads only.
But what you propose doesn't allow me to abandon jobs in the queue if the user is no longer waiting for their file. That's what I need.

Posted: Fri Feb 10, 2006 8:10 am
by BDKR
Honestly you guys, I don't understand why you'd want to use a Database for such an operation. This seems kind a Golden Hammerish.

Apache uses something called a Scoreboard wich is a shared memory construct. Here is a description taken from activestate.com talking about the Perl Apache::Scoreboard module...
DESCRIPTION

Apache keeps track of server activity in a structure known as the scoreboard. There is a slot in the scoreboard for each child server, containing information such as status, access count, bytes served and cpu time. This same information is used by mod_status to provide current server statistics in a human readable form.
Once again, this is an shared memory construct. Remember that Apache 1.3x uses forked child processes which means each of these processes are trying to access this area of memory independently of one another or some other orchestrating mechanism.

Now PHP supports doing this kind of thing and this algorithm would be a good deal faster then using MySQL for something it wasn't really intended.