Back to Feed
article 3m read

The agent gave my search table three shapes in a day, and nothing checked the old files

A small retrieval service guarded its saved data with create-if-not-exists while its text table changed shape twice in one day, so an older store would likely fail on missing columns, and the obvious drop-and-recreate could point saved vectors at the wrong rows.

Article DepthPractice

Flow

Adding a version guard, in order

  1. 1Save a version

    Store it with the data; note which model made the vectors.

  2. 2Check at open

    Compare the saved version with the one the code expects.

  3. 3Empty file

    Build the current tables and stamp them.

  4. 4Unstamped file with tables

    Refuse with a clear message about what is out of date.

  5. 5If migrating

    Keep row numbers, swap files by rename, write the version last.

The agent gave my search table three different shapes in one day, and the only thing guarding the data already saved was a check that a table with that name exists.

I found it reading the day's changes to a small search service the agent built for me. It keeps text in one SQLite table, the search vectors in a separate file, and a list that says which vector goes with which row. Each time it starts, it runs "create this table if it doesn't exist."

The table grew twice that day. First it got a checksum column so it could skip duplicate text. Then it got source and tag columns so searches could filter. Each time, only the create line changed. Nothing saved a version number, and nothing checked the old file against the new code.

"Create if not exists" only looks at the name. If a table with that name is there, it does nothing, even when the columns are wrong.

If an older file already has a table with that name, this line is skipped and the new column never gets added.

What an old file would do

I haven't tested this against an old file. From reading the code, three steps would ask for a column an older table doesn't have: the duplicate check, the insert and the search. Each one should fail. The server answers anything unexpected with a generic server error and the raw database message. So the caller gets a confusing fault instead of "your data is out of date."

Why rebuilding the table is a trap

Dropping the table and letting it be recreated sounds simple. But the vector file points at rows by number, and a new table starts counting again. Old vectors could then point at the wrong text, and search would return wrong answers with no error at all. The only built-in recovery is a reset that deletes everything. That's fine for a cache and painful if this is your only copy.

Refuse the old file before the first request

What this store lacked was a version number saved with the data and checked when the store opens. A new, empty file gets built and stamped. An unstamped file that already has tables gets refused. The vector file needs its own stamp too, along with which model made the vectors. If I migrate instead, the row numbers have to survive, files get swapped by rename, and the new version gets written last.

For now I'd have this one refuse an old store with a clear message, since I don't yet know if its text is the only copy.

Keep reading

Pick up a connected idea or branch into a nearby one.

5 paths forward
Related idea

The draft button checked for duplicates, then left a gap wide enough for a second one

The button that saves a delivery as a draft in the stores' inventory system checked first and wrote its record only after a slow outside call, so a second click could slip through. Now it claims the delivery in the database first, lets a partial unique index refuse the second claim, fixes a payload hash that couldn't see nested fields, and adds a narrow admin cancel for claims that get stuck.

via #databases · #reliability