Back to Feed
article 3m read

The same broken sessions came back for a retry on every summarizer run

The session summarizer found its work by looking for sessions with no record, so anything that failed came back on every run. Now a failure writes a stub record, the report skips stubs and counts them, and the promise in the warning got smaller to match what the code can actually do.

Article DepthPractice

Flow

What happens to a failed session now

The stub write is best effort, so a session can still come back if it doesn't land.

  1. 1Session fails

    Anything in the run throws, from the model call to the save.

  2. 2Stub saved

    Identity, counts, a failed marker and a short error, only if no record exists.

  3. 3Stub save fails

    The run logs a warning and moves on.

  4. 4Report skips it

    Stubs are left out of totals and averages.

  5. 5Warning shown

    The report page says how many sessions failed and were left out.

The session summarizer kept going back to the same broken sessions. Every run, it failed on them the same way and logged the same error.

The summarizer is part of a tool I'm building that reads recorded AI coding sessions. For each one, it asks a model for a few fields, like the goal and how it ended, and saves one record. Later, the records get rolled up into a report.

Work found by what's missing

There's no queue. The job looks for sessions that have no record yet and works on those. I like that, because there's no second list to keep in sync. But it only works if every try leaves a row behind. Mine didn't. When a session failed, the job logged it and moved on, so the next run found it again. The good sessions kept going through, so it looked fine. Meanwhile every run paid for a model call on each broken session, and the log filled up with repeats.

A failure that leaves a record

Now a failed session gets a stub record: who it is, the counts it already had, a "failed" marker, and a short copy of the error. The stub is only written if no record exists, so it never replaces a real one.

A failed session now gets a stub marked failed with a short reason, written only if no record exists.

Then every reader of that table has to know about the marker. The report step now skips stubs, so they don't get averaged in. The report page also shows how many sessions failed and were left out.

A promise bigger than the code

The first warning said these sessions would never be retried. But saving the stub can fail too, and if it does, the run only logs a warning. So "never" was too strong, and the wording got softened.

The PR review also asked for the failure count to follow the report's date range. That uses each session's start time, and the stub doesn't save one. If I'm reading that right, reports with a date range show zero failures.

Keeping the tool out of its own data

Some stored sessions come from the tool's own agent. If those get summarized, the tool ends up studying itself. Now the agent always labels its sessions with an owner marker, and the summarizer skips anything carrying that label.

Keep reading

Pick up a connected idea or branch into a nearby one.

5 paths forward
Related idea

Auth middleware would have logged everyone out the minute the auth server hiccupped

The SDK middleware turned every failed token check into a redirect to sign-in, so an outage on my side would have looked like a mass logout. I split the answer into yes, no and couldn't-check, and made the middleware answer an outage with a 503. In the same commit the browser client went the other way and now throws away good tokens when the server is unreachable.

via #reliability · #failure-modes