My retry table said done for answers it never saved

Flow

What happens when the answer is saved

1. 1Everything matches⌄

   The answer is saved and the token is wiped in the same write. The caller gets the answer.

2. 2Nothing matches⌄

   The save is refused. The code looks at the row to log why and never writes again.

3. 3Route returns an error⌄

   A refused or failed save becomes a server error, not a success.

My vendor API could mark an order delivered, fail to save the answer it gave, and still say "success". To the vendor, that success means "if this reply gets lost, send it again with the same key and you'll get the same answer". There was nothing saved to send back. A decision note the agent wrote called that save step best effort, and I didn't catch it. A later note took that back.

How the retry keys work

Vendors call my ordering system to change orders. Each call can carry a retry key. The server keeps one row per key, with a hash of the request, the saved answer, and an expiry time.

I built it in steps. First, each call claims its key before doing anything, by adding an "in progress" row. The database lets only one caller win. Next, every way out of a call saves an answer, errors included, so a key doesn't stay stuck for up to a day. That made the save step run all the time, so it had to be right.

The weak save step

The code remembered "I claimed this" in memory only. The save wrote to whatever row had that key, even if a newer caller had taken it over after the first claim ran out. If the row was gone, the save made a new one out of thin air. And if the save failed, the code logged it and still said success.

Giving each claim an owner

Now claiming a key makes a random token, stores it on the row, and gives it to the one caller that won. Saving the answer is a single conditional write. It only goes through if the key, the hash, the "in progress" mark, the token and the expiry all still match:

const changed = await store.setIf(
  row => row.key === key && row.hash === hash && row.ticket === ticket
      && row.pending && row.validUntil > clock(),
  { answer, ticket: none },
)
if (!changed) throw new NotSaved()

The token is wiped in the same write. If nothing matched, the code looks at the row to log why, but never writes again.

When the save fails, the route now returns a server error, not a success. A test checks this: the order is marked delivered, saving fails, and the vendor gets a 500. Other tests check that the owner's save works and that someone with an old token is turned away.

What's still open

• The tests fake the database, so I haven't proven this under real load.
• After that 500 the order really is delivered. Once the claim runs out, a retry can do the work again, and on a reissue I'd guess that's a second email.