Deploy forgot everything when it died, so now it keeps a ladder

Flow

When a test machine looks wrong

1. 1Compare⌄

   Check the installed tools against the reviewed version.

2. 2Matches⌄

   Touch nothing and move on.

3. 3Busy with another run⌄

   Wait and check again, up to a fixed limit.

4. 4Free but wrong⌄

   Reinstall the reviewed tools, then run the self-check.

5. 5Still busy at the limit⌄

   Stop the deploy and leave the other run alone.

Interrupted, the production deploy had no memory of where it stopped. It's one command for the ordering system, and it does a lot of steps in a row: test the code, get it signed off, copy it over, check the database, try it out, switch it live, then clean up. If it stopped halfway, the only thing it knew how to do was start again from the top.

So I had two bad choices. Run the whole slow test pass again for code that had already passed. Or fix up the half-done state by hand and hope I remembered what had really happened.

Pin one commit first

You can't pick up where you left off if you don't know what you were shipping. The old deploy shipped whatever my folder held, and only checked that my commit was somewhere in the main branch's history. An old commit could still pass.

Now the command fetches main once and requires my commit to match it exactly. If it doesn't, it tells me why: I'm behind, I have commits I never pushed, or the two have split. Then it builds the release from a clean private copy of that one commit. My unsaved edits stay out of it, and a push that lands halfway through can't change what's being shipped.

Keep a record per commit

The production machine now keeps a record per commit of how far the release got. Run the same command again and it reads the record and skips what's done. The pattern is simple.

Each step runs only if the record says the release hasn't reached it yet, and the record moves forward when the step finishes.

One rule matters a lot. The release's sign-off has a deadline. Before the code goes live, an expired sign-off still stops everything. After it's live, the deadline no longer blocks finishing, because stopping a live release halfway is worse than finishing it late.

This only works if every step is safe to run twice. I believe that's true, but I haven't proven it for every step.

Read the database before the ticket

Database problems used to show up at the end, after the long test run. Now a check runs first that only reads. It looks for broken migration files and tool mismatches, then asks the production machine to confirm its database is healthy and matches what's expected. It never changes anything. The real migration still happens later, in one place.

Wait for other people's runs

The deploy also checks that the test machines have the right tools installed and repairs them if not. If a test run was already going, the old code just quit. Now it waits a while, checks again, and never repairs a machine in the middle of someone else's run.

That waiting showed a dumb timer. The release had a time goal, and waiting in line counted against it. Now waiting is timed on its own clock, and the release clock starts when the real work does.

What I don't know yet is how long that line usually is. Now that waiting has its own number, I'll find out.