Flow
Running tests without touching the real config
1Private copy per test
An environment variable tells the app where its config is.
2Update the old tests
They stop writing to one fixed spot.
3Fallback: one run at a time
A lock, with the saved copy kept on disk.
4Fallback: move the tests
Run them somewhere that has no real users.
The chore-scheduling app has an admin page with a button that runs the test suite on the live server. I didn't think hard about that until I read what the agent built for it.
Tests that write the real config
I didn't read the test scripts themselves, but the runner the agent wrote only makes sense if some of them write their own config file into the app folder. With a signing-secret test in the suite, one of them probably breaks the secret on purpose to check that sign-in tokens get turned away. On my laptop that's harmless, but on the server that file is the real configuration.
The runner and the button
The agent wrote a runner that day. It runs each test script in a separate process, saves the configuration in memory first, and writes it back after every test. At the end it compares the file with the saved copy and reports whether they match. For one person typing in a terminal, that's good work.
Then the agent hooked the admin button up to the same runner. The old button restored nothing at all, so this looked like an obvious upgrade. The page now says the run leaves your settings alone.
That's true at the end of the run. It isn't true during it.
What the site sees mid-run
While a test runs, the config holds test values. If the site reads that file on every request, a real user can be served with test settings. If my guess about the secret test is right, their sign-in check could fail. I haven't checked whether my server gets the secret from somewhere else first. If a test deletes its file, there's a short gap with no config at all.
Putting things back afterwards only works if nobody else looks in the meantime. A web server looks all the time.
Two runs at once
The button greys itself out, but only in that one tab. Open two tabs and, as far as I can tell, both runs go.
Run B saves while run A has the file gone, so it saves nothing, and later puts that nothing back, which deletes the real config.
Run B saves "no file" as the original, puts that back, and then checks against its own wrong copy. It reports green.
When the run gets killed
The saved copy only lives in the process memory. If the server kills the run partway through, nothing restores the file, and the test configuration just stays in place.
What I'd build instead
Give each test its own private copy of the config, and tell the app where to find it with an environment variable. Then nothing is shared and there's nothing to restore. Two runs can't clash, and a killed run leaves a stray temp file, not a broken site. The catch is that my old tests have to stop writing to one fixed spot.
If that's too much work right now, there are two fallbacks. One is a lock so only one run can go at a time, with the saved copy kept on disk. The other is to run the tests somewhere that isn't serving real users.
Until one of them is in, that button is a gamble every time I press it.