Backups that only got a second copy on Sundays

Flow

What the copier checks, copies, and records

1. 1Right source⌄

   The landing zone is a real mount of the right filesystem, not an empty folder.

2. 2Locked-down access⌄

   Owner-only SSH key and a pinned host key for the NAS.

3. 3Right destination⌄

   Marker file present, expected pooled filesystem, repository opens.

4. 4Copy each repository⌄

   Skip locked ones, time-limit each copy, record every failure.

5. 5Close out⌄

   Integrity check, snapshot count, receipt. Any failure fails the run.

Between Sundays, the newest backups from my main Proxmox box lived in one place: a landing zone on my backup VM.

It's a good first copy. Clients can add to it but can't delete from it, and the server owns cleanup. But the only other copy ran on Sundays. If the landing zone had died on a Saturday, I'd have lost most of a week. My NAS had an encrypted repository ready for this. Nothing filled it on a schedule.

So I spent a day on backups. Three jobs, two fixes, and some gaps still open.

Getting the right things into the first copy

The first job is for a web app that keeps its data in MySQL. If you copy a database's files while it's writing, you get a backup that won't restore. So the job dumps every database in one transaction. It writes to a temp name and renames the file only when the dump is done, so a failed run leaves the old dump alone. Then it backs up the dump and a short list of app and config folders. It takes a lock so two runs can't overlap, it doesn't catch up after downtime, and it never deletes old backups.

The second job is for my virtual router. Its config lives inside the guest, where the host backup can't see it. A script on the host asks the guest agent for the config, checks that it looks right, saves it with a checksum, and moves it into place just before the host backup runs. The host backup now includes that folder.

The copy that has to prove where it's writing

The third job is the one that matters. Twice a day, a copier on the backup VM copies every repository for this box from the landing zone to the NAS. The second run is a retry.

Before it copies anything, it checks a few things:

• the landing zone is a real mount of the right filesystem, not an empty folder
• the SSH key is locked down and the NAS's host key is pinned
• a marker file on the NAS says this is the right place
• the NAS repository can be opened

The failure I'm guarding against is dull. A disk doesn't mount, the job writes into an empty folder, and it reports success. Those checks turn that into a refusal.

Inside the loop, it skips any repository that's still locked by a backup in progress. Each copy gets a time limit, and failures go into a list. At the end it checks the NAS copy, counts snapshots, and writes a receipt. Any failure fails the whole run. Nothing in it deletes anything.

One gap I can see: if the final check fails, the script stops before writing a new receipt. The old receipt stays, looking fine. So check the receipt's age, not just what it says.

Two checks that couldn't do their job

The router job wrote its checksum file while the config was still in a temp folder, so the file pointed at the temp folder's path. Then the folder got renamed. The checksum pointed nowhere. The follow-up commit records just the file name.

The second one is better. All three installers had a line meant to stop if a file had Windows line endings. It started with ! to flip the test's result, and Bash's stop-on-error mode ignores any command flipped that way. So the guard never stopped anything. The shell linter caught it. Put the test in an if and exit on purpose:

if file "$conf" | grep -q CRLF; then exit 1; fi

What I still don't have

The NAS pool isn't a mirror, so one dead disk can still hurt it. Nothing from that day pulls a snapshot back off the NAS, so I can't promise this copy would come back.