Flow
Where "running" lives
Moving the flag out of memory is what lets a second copy of the app see it.
1Before
In the old copy's memory, invisible to the new copy.
2Symptom
New copy says not done, page starts the build again.
3After
A lease file in the build's folder that any copy can read.
4Bonus
The saved refresh and progress let a takeover continue instead of restarting.
One order build turned into two when a dev deploy swapped the app halfway through it.
The build is a slow job. It pulls fresh stock and sales from the stores, plans the order and saves a draft, and the web page keeps asking for progress. My dev deploy starts a new copy of the app, sends traffic to it, gives the old copy a short time to finish, then stops it.
The problem was where "this build is running" lived. Only the old copy knew, because it kept that fact in memory. When the page asked the new copy, the new copy said the build wasn't done. The page took that to mean nothing was running and started the build again. The second build redid the slow refresh, hit a file the first build had already written, and wrote "failed" over the progress.
The agent and I had a zero-downtime deploy and a long job, and we never asked what happens when they meet. A deploy is a restart, and this job could outlive one.
Where the running flag lives now
The flag is a small lease file in the build's folder. A build creates it only if it doesn't exist yet, so only one process can hold it. The holder touches it every few seconds, and it notes which process and port own it. Any process can read it, so either copy of the app now says "running". A second request waits for the first build to finish and reuses its result. If the holder dies, its file goes stale and the next request takes over.
The slow refresh is saved, so a takeover reuses it while it's still recent. Progress carries on instead of starting over. The deploy also checks the lease files and waits a while longer for builds on the copy it's about to stop.
Tests with fake processes cover this. A real deploy in the middle of a real build hasn't happened yet, and a process that freezes past its heartbeat could lose its lease while still working.