Flow
A pipeline that reports what happened
Order of operations from agent run to reviewer.
1Run the agent
Let it edit the box. Keep its words, but don't act on them.
2Ask the tree
Ask version control what changed. That list is the change set.
3Check the files
Each file must sit in the allowed folder and match the format.
4Give the result a status
An answer you can't read scores as unknown, never as perfect.
5Save both lists
Store what the agent claimed beside what changed, flag any gap, and keep the whole log.
6Show the reviewer
The real files come first. The agent's words are notes, not a switch.
The agent spent the whole run writing reports about its work. A report, a summary, an index of its summaries, and more. It ended with a happy wrap-up. It never printed the small answer the pipeline asked for.
So the pipeline made one up: don't act, zero confidence. It logged that no files had changed, though new files sat right there. Then it called the run complete.
The log of that run was saved next to the code. It came from an older version, so it may not match what shipped. But it made me reread the code with one question. When this thing decides something, whose word does it take?
Three records of one run
A run leaves three records. There's what the model says about its work. There's the set of files that really exist at the end. And there's the record the system saves. Only the files can't be talked into anything. So I checked which record each decision reads.
The timer trusts the model's numbers
One part runs on a timer. It asks a model for a list of fixes, each with a priority and a confidence score. It starts in a safe dry-run mode, which is good. When that's off, it installs the fixes the model rated high and confident.
But the model picks both numbers, and the prompt tells it which numbers pass. Worse, missing values get filled in, and a confidence of zero gets treated as missing. "Not sure at all" turns into "half sure".
Then there's the failure case. If the model's answer can't be read, the parser returns an empty list. The health score starts at the top and drops for each high or medium priority fix. So an unreadable answer scores as perfect. If you build this, give the result a status.
A result that isn't ok gets no score at all rather than a perfect one.
The sandbox publishes on a text match
The other part runs an agent in a throwaway box with a short-lived token. Some of it is solid. The files it uploads come from the version control diff, which is the right source.
The choice to publish at all is not. The agent is told to end with a flag saying whether it made anything. The code never reads that flag. Instead it searches the agent's output for text that looks like a skill file path. If it finds one, it commits and opens a proposal.
That search and the diff never get compared, so:
- A path the agent only mentioned makes an empty commit fail, and the run is marked failed.
- A file it wrote without printing the path gets no proposal at all.
- The proposal can describe a different number of skills than it uploads.
The run record keeps only the start of the log. And the cleanup step that deletes the box is commented out.
The same day in the history
The repo got around twenty one-file commits that day, each adding a skill file. Their messages match the uploader's format, so I'm fairly sure that's the source. At least one security topic landed twice, in two different shapes.
Asking the tree first
Ask version control what changed. Check those files against an allowed folder and a format. Treat the agent's words as notes for the reviewer, not as a switch. Save both lists and flag any gap. Keep the whole log.
And put the cleanup step back. As the code stands, every finished run can leave its box behind.