Flow
How I want missing scores to show up
1Every score names its origin
Judged, estimated by a rule, or missing.
2No friendly defaults
A blank stays blank instead of becoming zero or a nice number.
3Averages use judged scores only
Stock turn values and fixed numbers stay out of the math.
4Coverage printed next to the mean
The report says how many sessions and turns the judge really read.
5A third gate result
Too little judged means can't tell, not pass.
6Tests that turn the judge off
A failing judge, an empty judge and a wrong-length answer, each checked.
The grader I built for my coding agents has a bad habit I found by reading it closely. When its judge fails to answer, it makes up a decent grade and saves it next to the real ones. You can't tell them apart.
The grader came in as one big commit. It takes finished agent sessions and asks an AI model, the judge, to score three things: was the work correct, was the code good, did the task get done. It also scores efficiency by comparing each session to similar past ones. Then it mixes the four into one overall score, saves it, averages the run, and checks the average against a pass mark.
The judge is a call to a model over the network. It will fail now and then. What counts is what gets saved when it does.
One failure, two fake inputs
The three judge calls run together. If one fails, all three answers get thrown out, even the good ones. Correctness and code quality then come from simple rules. Task completion has no simple rule, so it gets a fixed number set a bit above the middle. The same number is used when there's no judge at all. Satisfaction is just a copy of task completion, so one made-up number fills two of the four slots.
A skipped score, two defaults
If the judge answers but skips a score, the defaults don't agree. Correctness and code quality become zero. Task completion becomes that friendly fixed number. Same empty answer, opposite results.
Every one of these rows is saved with the same label as a real judgment. The averages count them all the same.
Guessed turns
A second check scores small groups of turns inside a long session. If that call fails, each turn gets stock values for understanding and coherence, a progress guess based on whether a tool ran, and a precision guess where using no tools counts as perfect. If the judge returns the wrong number of turns, some turns get dropped or the whole group gets guessed.
Those guesses go into the same math as real scores, and the math can't spot them. A flat line never shows a sudden drop. The stock values never sit low enough to count as bad. So a session the judge never read comes out calm and a bit above average. The judge's own prompt even asks it not to bunch up in that range.
The efficiency score has a smaller version of this. With no past sessions to compare against, it scores against a made-up baseline built from zero samples.
The one place that gets it half right
The code that compares agents marks its fallback rows with a different label and flags them in its log. I haven't checked whether anything that reads the scores pays attention to that label.
What a score should carry
Each grade would hold a value that may be empty and a label saying whether it was judged, estimated or missing.
I want every score to say where it came from. Averages should use judged scores only, with a count of how many were judged printed next to them. A run where too little was judged should come back as "can't tell" instead of pass or fail.
None of the tests that came with the grader ever turn the judge off. Every guess path in it is untested.