I couldn't get past one thing,
say we have 100 instances of the script that is running through cron
it seems up until the 1st instance did not finish running entirely
that the database does not get updated therefore for an instance to finish
it could take 10 minutes and all other 99 would be trying to work on the same
position as the 1st one
we need to somehow free this up by being able to update the database and then initiate
an instance
does that make sense?
I have a position database,
position 0
I start running a script through cron and I update the position to say 200.
Then if I were to run the same script in parallel multiple times I am unable to read the position updates because for some reason up until an instance has completely finished running (which could take up to 10 mins. run)
the database doesn't get the update
Thanks
Cron job, I can't get past waiting for an instance to finish
Moderator: General Moderators
Re: Cron job, I can't get past waiting for an instance to fi
The best way to deal with (what seems to be) the problem is to use a master/slave setup: one single process acts as the master and fetches the various pieces of work, then sends it to one of many slave processes waiting for work. That way there's only one process pulling from and updating the database - at least in terms of figuring out what work to do - at a time.
The next best that I know works as long as you don't have very high load on MySQL (I don't know for other database systems): do a SELECT to figure out what thing you should work on that's not already "claimed", mark the thing as claimed under the condition that it's not already claimed, check that it updated properly, and work on it. If it didn't update then you hit a race condition where another process found the same item but managed to claim it first.
The next best that I know works as long as you don't have very high load on MySQL (I don't know for other database systems): do a SELECT to figure out what thing you should work on that's not already "claimed", mark the thing as claimed under the condition that it's not already claimed, check that it updated properly, and work on it. If it didn't update then you hit a race condition where another process found the same item but managed to claim it first.
Code: Select all
SELECT * FROM work WHERE status = unclaimed LIMIT 1Code: Select all
UPDATE work SET status = claimed WHERE id = what you got from before AND status = unclaimed