Flow
Which failures get a retry
1Disk full during an upload
Remove the partial file, clean up, try one more time.
2Failed test or check
No retry. Fix the cause.
3Bad receipt or signature
No retry. The proof is wrong, not the disk.
4Database write or promotion
No retry. It may have half happened.
5Still full after cleanup
Stop, print the numbers, resume from the same step later.
Failed uploads got past the upload cleanup. If an upload to my release box died partway, the half-written file would stay behind. The cleanup code got the file's name from the upload function, and a function that throws gives you nothing back. But the name was built from a token I already had, so now the cleanup lists every possible name before the upload starts.
const leftovers = [`incoming-${token}.pkg`, `incoming-${token}.pkg.gz`]
Where the space went
The box gets releases for the ordering system. Each release leaves several big copies on disk. The old cleanup removed old build folders and backups, but the big archive kept for every release was never removed, and a missed upload would sit beside it. Nothing gave that space back.
When the disk fills during a deploy, the easy move is to delete big old files. On this box those include the rollback copy, a release still being checked, and the signed proof that a release was reviewed. I don't want that done in a hurry by me or by an agent. So cleanup is now a planned job.
The releases it won't touch
The job first decides what it must keep: the live release, the one before it, anything unfinished, the newest few finished ones, and any record it can't read. If it can't tell which release is live, it stops.
Then it only looks at a short list of known payload files and old failed uploads with the exact upload name. Proof files always stay.
Check every file
Each file must not be open by a running program, must not be tracked by git, must be a plain file with the right owner, and must not have changed since the plan was made. If any check fails, the file is skipped. The default is a dry run.
A lock that can't get stuck
The old lock was a folder that stayed forever if a run crashed. The new one is a kernel lock that goes away when its owner dies.
Ask for the right space, retry once
Before uploading, the deploy measures its files and asks for a reserve plus several times their size. It also trims the disk from the host and checks the host pool. If there still isn't enough room, it stops and prints both numbers. If an upload still fails because the disk is full, it removes the partial file, cleans up, and tries one more time. Any other error stops the deploy.
A bash trap
Bash ignores set -e inside a function used as an if test, so a failed step could look like success and skip rollback. Each step now returns on failure.
Of everything here, that one scares me most: a broken install that says it worked.