Back to Feed
article 3m read

Backup system the agent hardened could write a backup but not restore one safely

The agent built an encrypted, signed, two-copy backup system with no tests. Reading the restore path showed a loose success flag, a signature nobody checks, truncate-before-load, and comma-split CSV, plus what I want restore to do instead.

Article DepthPractice

Flow

The restore path I want

  1. 1Check

    Signature and every file's checksum, before touching the database.

  2. 2Parse

    Read each CSV with a real parser, not by splitting on commas.

  3. 3Stage

    Load into a side table and compare the row count with the manifest.

  4. 4Swap

    Replace the live rows inside one transaction, so a failure changes nothing.

  5. 5Rehearse

    Run the whole path into a scratch database on a schedule.

The agent wrote my app a backup system in one go. It looked strong: encryption, a signed list of contents, two copies, checksums. But it came with no tests, and nothing showed a restore had ever been run. So I read the restore code as if I needed my data back right now.

Making the backup is the good half. Each run dumps the whole database, saves every table as a CSV file, writes a manifest of row counts and checksums, signs it, encrypts it all, and copies the file to two places.

The weak spot is the word success. A job counts as a success if just one of the two copies checks out. If one table fails to export, the job still says success. And the full dump and the CSV files are taken one after the other, so they don't hold quite the same data.

The signed manifest is never checked when you restore. Only the hash of the whole file is. So a missing table or a bad CSV goes through.

Then the restore order. To restore a table, the code empties it first, saves that, and only then loads the CSV. If the load breaks, the table stays empty. The empty step also clears any table that points at it. Restoring one customer's data reads the CSV by splitting lines on commas, so a quoted comma before the tenant column can make it drop a row, then deletes their rows and loads what it picked out. A table whose export failed is skipped with no warning.

What I want the restore path to do:

  • check the signature and every file's checksum before touching the database
  • load into a side table, compare the row count, then swap inside one transaction so a failure changes nothing
  • parse CSV with a real parser
  • call a backup a success only when both copies and every table made it
  • restore into a scratch database on a schedule

Retention matters too. It keeps the newest good backup per day, week and month. Since a partial backup counts as good, it can push out a complete one.

Two smaller bugs. The scheduled script sends the backup type where the server doesn't look, so every run is labelled daily. And unless two settings are filled in, both copies land in folders that are almost certainly on the same disk.

Keep reading

Pick up a connected idea or branch into a nearby one.

5 paths forward