Flow
Every new limit, from the slot to the alert
Start at the slot, end at the alert.
1Slot deadline
A stuck task fails and gives its slot back
2Fast lane
Small reads skip ahead and can say busy
3Job budget
An overrun job fails and says what it was doing
4Item limit
A stuck item names its step, the loop moves on
5Progress alert
Fires when ready jobs sit too long
Testers on the dev box watched store pickers spin for minutes. No error, no log, nothing until the app restarted. Earlier in the week, a nightly job on prod had hung and held up every other job for days. Both had the same cause: a wait with no clock on it.
Every call to the upstream inventory API had a timeout. But the calls wait their turn in one shared line, one at a time. The timer only started when a call reached the front. If the call at the front never finished, it kept its slot, and nobody behind it ever got a turn. Their timers never started.
The store list is shared, too. One request feeds every picker on the page, so one stuck read froze them all.
The nightly job was the same story one level up. It took its job and then waited on something that never answered. No CPU, no traffic. The process was up, so nothing looked wrong. Prod had a single worker, so everything else waited. The job library has no time limit per job at all.
Clocks on the parts nobody timed
First, a deadline on every slot. Each task in the line now gets a hard limit a few seconds past its own timeout. If it's still running when that passes, it fails with its own error, the log names it, and the slot goes back. Make sure the slot can only be handed back once, or a late finish will hand it back twice.
Second, a fast lane for people. Small reads someone is waiting on now skip ahead of big background reads. They also get a wait limit. If they can't start in time, they fail with a "busy" error, so the page can say so. Keep "busy" and "too slow" as different errors. One means try again soon; the other means the upstream is struggling.
Third, a budget for every background job, a few times longer than its slowest good run. When it runs out, the job is told to stop, the log says what it was doing, and the job fails so the worker is free again. Long jobs that loop over items should also give each item its own limit and note which step it was on.
Fourth, an alert on progress. Every few minutes, ask the database how old the oldest ready-but-unclaimed job is. If that age keeps climbing, something is stuck, even when the process looks fine. Add a second alert for when that check itself fails.
Things that bit me
A deadline doesn't stop the stuck work. JavaScript can't cancel it. It just stops waiting and frees the slot, and the stuck work keeps running in the background until it ends.
Don't fix a stall by adding more workers. It hides the hang and uses more memory. My dev worker ran out of memory around the same time.
I still don't know what the nightly job was waiting on. Now, if it happens again, the log will say which step it was stuck in.