Flow
Weak spots in the evidence behind a rewrite
1What active means
It may only mean the file was merged before the session started.
2Yes-or-no counting
One small slip counts like a ruined session, and the count stops at a page size.
3Very short trends
The stored significance value is ignored, so noise can flag a fine file.
4The reason sentence
First and last numbers only, so a bumpy history reads as a steady decline.
5What the agent sees
A short summary and the start of each failing session, not where it went wrong.
Insights showed one of my agent instruction files getting worse, but couldn't do anything about it. So I built the part that does. It finds the files that are slipping, hands each one to a coding agent with the sessions where it seemed to fail, and saves the agent's rewrite as a new version that points back to the old one.
It's in. Then I reread the diff, and the parts I liked most stand on the weakest evidence. Nothing in the change shows a rewrite coming out of it yet, so this is about the design, not results.
How the loop runs
Every session already records which instruction files counted as active and what kinds of trouble came up. A file gets picked when it's been merged, its trend is marked as getting worse, and its history passes a very low minimum. Then I pull its misses: recent sessions where the file was active and the trouble it was meant to stop happened anyway.
The agent gets the file, a few of those sessions and a one-line reason. It's told to figure out why the file failed before editing, keep what works, and add a wrong-way and right-way example. The new version is saved with a link to its parent, and the parent's record is left alone, so I can compare the two later.
Where the evidence is thin
These are the spots where I trust my own numbers least:
- What "active" means. The test for the lookup says it returns every file merged before the session started. If that's all it is, a miss just means the trouble showed up after the merge.
- Yes-or-no counting. One small slip counts like a ruined session, and the count stops at the query's page size.
- Very short trends. The stored significance value is ignored, so a couple of noisy numbers can flag a file that's fine.
- The reason sentence. It compares only the first and last numbers, so a bumpy history reads as a smooth decline. The agent gets the raw numbers too, but I'd bet it leans on the sentence.
- What the agent sees. A short summary and the start of each failing session. I'd guess the real mistake usually comes later.
The reason sentence is the easiest to show. It's basically a two-point comparison.
It just says the numbers got worse from the first one to the last one.
I'd rather send the range, the latest sample size, and whether the numbers ever turned around.
Where the run itself can go wrong
The new version link is stamped on everything the run makes, so an unrelated extra suggestion gets labeled as a rewrite too. If that stamp fails, it only logs a warning.
The trigger says "started" right away and does the work afterward. If the work fails, the error goes to a server log, and the page keeps saying "started." Nothing stops two runs on the same file at once, and I think both would get the same version number. The target is handed to the agent through a loosely typed field, and if that ever breaks, the agent runs its normal job instead.
Tests stop before the loop does
The tests cover the queries and the new fields on a real database. The link test sets the parent by hand. Nothing runs the trigger or the whole loop.