Flow
Keeping restarted servers alive
1Background launch from a one-time service
The server starts, then gets cleaned up when the service finishes.
2One line: stop only the script
Servers survive, but still count as part of the recovery service and nothing restarts a crash.
3Own service per server
A restart rule keeps it up, and the script just asks it to start.
4Check it is really up
Look at how long the server has been running, not at the recovery log.
The box that runs my local models could not come back on its own after a reboot. The model servers had been started by hand. The dashboard had a service, but it ran the wrong command and kept failing, so a hand-started copy did the real work. One power cut and I'd be rebuilding it all from notes.
A timer and one careful script
I set up a timer that runs shortly after boot and then every few minutes. It starts a short one-time service, which runs one script. The script checks each thing first and only acts if it's broken. It won't start a second server on a port that's stuck. It checks that files exist before it launches anything. It runs my programs as my user. It leaves alone a gateway that isn't mine. And one broken step never stops the rest. I also fixed the dashboard's service so it runs the right command.
Why the server wouldn't stay up
The script started the model server that had no service of its own in the background, waited for it to answer, wrote "recovered" in the log, and finished. The trouble is what systemd does when a one-time service finishes. It cleans up every process the service started. That includes a server running in the background, even one started with nohup.
So each round would start the server and then lose it a moment later. The next round would find it gone and start it again. Every log line would say things were fine.
I don't have a log of it actually eating a server. What I have is the follow-up, the same day, and it's one line in the service file:
KillMode=process
That tells systemd to stop only the script when it finishes and leave its children running. Now the next round sees the server healthy and moves on.
Still owed
The servers still count as part of the recovery service, which is confusing. Nothing restarts one that crashes until the next round. And a server that starts but never gets healthy now stays put and blocks the next try, so I need to watch for that in the log.
The real fix is for each long-running server to get its own service with a restart rule. Then the script only has to ask for it to start, and a crash doesn't have to wait for the next round.