Flow
How a late save gets turned away
1Claim
A new claim bumps the number and sets the expiry by the database's clock.
2Heartbeat
Slow steps renew before, during and right before the save.
3Lease runs out
Another worker claims the job and gets a number one higher.
4Late save
The old number matches no row, so nothing changes.
5Stop
The late worker marks the run blocked, not failed, and leaves it alone.
The lease on my supplier setup runs checked a name tag, and the worker with the tag got to say when its own time was up.
One job, one worker at a time
I have a long job that sets up a new supplier, step by step, and saves a receipt after each step. If a worker dies, the next one picks up where it stopped. Only one worker should run it at a time. A worker claims it with a lease: its name on the row plus an expiry a couple of minutes out.
The expiry is what lets someone take over after a crash. It also means a worker can stall, lose its lease, wake up, and still think the job is its own.
Two clocks and a name tag
The expiry used the worker's own clock, and the next worker checked it with a different clock. They could disagree.
Saves only asked "is my name on the row?" They never checked the expiry or which claim this was. A worker could even renew a lease that had already run out.
Slow steps didn't renew while they ran. A long step could lose its lease halfway, and the error said "busy" when the truth was "you lost it."
I doubt this ever broke anything, since each attempt gets a random name. But I don't want luck guarding purchasing.
Every claim gets a new number
Every new claim now bumps a counter, called a fencing token, and sets the expiry from the database's clock. The worker keeps its number. Every phase save must match all of this, in one statement:
WHERE owner = $1 AND fence = $2
AND lease_expires_at > clock_timestamp()
AND phase = 'previous'
If no row changes, the worker lost and stops. The phase check also stops a worker from skipping or repeating a step.
Slow steps now run with a heartbeat. It renews before the work, every so often during it, and once more before the save. If a renewal fails, that failure wins, even if the work looks fine. A lapsed lease can't be renewed. You have to claim again and get a new number.
A worker that loses marks the run as blocked, not failed, and leaves it alone.
A lease that expires in a blink
One test uses a real database. Worker A gets a very short lease, the test waits past it, and worker B takes over with a number one higher. A's save is refused, B's goes through, and there is only one receipt. Nothing new shows up in the purchase or setup tables. Other tests use a fake clock to check the heartbeat timing.
A stale worker still keeps working until its next heartbeat or save tells it it lost. That work is wasted, and none of it lands in the run.