Back to Feed
article 3m read

Reorder approvals that lived in memory while the job outlived the request

The slowest assistant job, building a reorder, ran inside the request on an in-memory approval token, so a timeout or restart left nobody knowing whether a draft order existed. Now apply writes a synced journal entry before it answers, workers hold fenced leases, an old approval can never restart a job, and an expired lease reads as interrupted rather than failed.

Article DepthPractice

Flow

Claim outcomes for an existing operation

  1. 1Different operator or arguments

    Refuse: the identifier is bound to its original request.

  2. 2Already succeeded

    Return the stored result; start nothing.

  3. 3Same approval grant seen before

    Return current status; a duplicate apply is harmless.

  4. 4Live lease held by another worker

    Return current status; do not start a second run.

  5. 5Failed or lease expired, new grant

    Start attempt N plus one with a new worker token.

  6. 6Lock held by someone else

    Refuse as busy; retry the same identifier.

The slowest thing the AI assistant can do in the ordering system is build a reorder. It pulls fresh stock history, plans the order, and writes a draft. That pull was slow enough that I spent part of the same day speeding it up.

And I was running the whole thing like a quick write.

Preview, then apply

My assistant uses a small tool bridge to make changes, and every change is two steps. Preview does a dry run and returns a token. I approve the plan, and the assistant calls apply with the token. The token only works for those exact arguments, it expires fast, and it lives in memory. I kept all of that.

The trouble was that apply ran the job while the request waited. If the request timed out, the assistant only knew it had asked. If the bridge restarted, the record of the approval was gone. And the reorder had the same short time limit as a quick read.

Apply saves a record, then answers

Apply now journals the job before it answers.

The job gets its record first, a worker starts only if this call made it, and the reply is just queued plus the ID.

The operation ID is part of the preview, so I approve it too. The record holds who asked, the exact arguments, a hash of the approval, an attempt number, and a lease, which is a claim that runs out unless the worker keeps renewing it. It is saved to a temp file, synced, and renamed into place. A lock keeps two claims from racing. Only then does the assistant hear "queued", and it polls that ID.

What a claim does

The claim mostly says no:

  • Same ID, different arguments or person: refused.
  • Same approval again, a job that already succeeded, or a live lease: you get the current status, and nothing starts.
  • The lock is busy: refused, so retry with the same ID.
  • Failed or expired, with a new approval: a new attempt starts.

The worker has a secret token, and every update has to show it. A worker that lost its lease can't overwrite the new one. An old approval can never restart a job. Going again means a new preview and a new yes.

What status says when the lease lapses

A separate read-only status tool reports progress. It only answers the person who started the job. A lapsed lease reads as interrupted, not failed, since the work may be half done. Status also checks the job's result files against the real database. If they disagree, it says to reconcile first, not to retry. Retrying there could mean a second order.

The reorder also got a longer time limit, a clean stop-then-kill, and a startup check that the record folder can be written.

What the token can't undo

The worker token narrows the window where two workers both think they own the job. It can't take back anything a subprocess already wrote before it was stopped. That part is still on the status check and the real order records.

Keep reading

Pick up a connected idea or branch into a nearby one.

5 paths forward
Related idea

New users logged in to nothing because sign-up waited so long to pick a workspace

Workspace creation waited for a later webhook so invited people wouldn't get a spare one, and self-serve people could log in before it fired. Now the creation handler asks the identity provider for memberships on the spot, a later event repairs misses, and the session lookup retries, but the comment that justifies it contradicts the one it replaced and nothing tests which is right.

via #reliability · #distributed-systems