Flow
How a missing time became a pass
1Agent call
Runs with no timer around it, only a tracing tool that keeps its time to itself.
2Record
Saved with the time left blank.
3Speed rule
Reads the blank as zero and passes.
4Unit test
Uses a fake record with a time filled in, so it stays green.
5Pipeline test
Runs the real tool with only a text rule, so the speed rule is never tried.
Agents that are too slow should fail under a rule in my eval harness. It can't. A very slow agent still gets a pass, and the saved result says the run took zero time.
Nobody started the clock
The harness runs each test case through the agent, saves a record of what went in and came out, and then checks that record against a list of rules. The speed rule reads a time from the record. But nothing ever writes that time. The code that calls the agent has a to-do note about timing it, and then just calls it. A tracing tool already wraps the call and knows how long it took, but that number never makes it into the record.
Empty turns into zero
When the time is missing, the rule uses zero. The bug came down to one default.
A missing time becomes zero.
Zero is under any limit, so the rule passes every time. The saved result shows zero next to the limit, and it looks like a real reading. A fast time looks like good news, so nobody stops to ask about it.
Both tests missed it
The unit test makes a fake record with a time already filled in. It checks that a loose limit passes and a tight one fails, and both work. But that fake record is the only place the time ever gets set.
The full pipeline test uses the real runner, but only with a text rule. The speed rule never runs through the real pipeline, which is the only place the bug can show up. What's missing is one test with a slow agent that expects the speed rule to fail.
Four more places that say yes to nothing
The same habit shows up across the harness:
- An empty search string counts as found in any text.
- The JSON check passes anything that isn't text.
- The score helper gives full marks when there are no results.
- A suite with no rules passes every case.
Each choice made sense alone. Together they turn a missing input into a pass.
Three answers instead of two
I want the rule to have a third answer, "couldn't check," for when the time is missing. The runner should time the agent call itself, with a clock that can't jump backward, and store that number on the record. Then a suite with no timing shows up as unchecked instead of fast.