A latency gate that passes when the speech engine never even loads

Flow

A gate that starts from what it expects

1. 1Declare ownership⌄

   Each suite lists the measurements it is responsible for.

2. 2Build the expected list⌄

   Blocking targets for that suite form the list the gate must see.

3. 3Compare⌄

   Anything on the list with no data fails and is printed by name.

4. 4Refuse an empty check⌄

   Zero blocking targets checked counts as a failure.

5. 5Check the config⌄

   Entries that match no real measurement are rejected when loaded.

6. 6Print the counts⌄

   Expected, checked, missing and failed, every run.

If the speech engine in my voice assistant fails to load, my speed check still says the build passed. I found this by reading the test code that came in with the first big import of the repo.

Two kinds of budget

My code has two speed budgets, and only one of them can actually stop a build.

The first one runs inside each request: each step of the voice pipeline gets a slice of time, and a timer measures the step when it ends and logs a warning if it ran long. The comments say a slow step must switch to a cheaper path, but nothing in the code does that. The timer only records what already happened, which is useful for debugging, but it can't prevent anything.

The second one is the gate. It's a check in the test harness that runs after a test suite and compares the results against a list of targets. Each target names a measurement, sets limits for the typical time and the slow times, and says whether breaking it should block the build.

How the gate goes green on nothing

The gate goes through its targets one by one and looks up the matching measurements by name. If it can't find them, it skips that target without recording anything. At the end it asks whether every blocking target it checked passed. If it checked none, the answer is yes.

The verdict is a pass whenever every blocking target it checked passed, which is trivially true when it checked none.

My harness has three ways to lose the numbers:

• The speech engine fails to load. The suite logs it, moves on, and ends with nothing to check.
• Every request to the router fails. Failed requests are thrown out, so there's no data left.
• Targets come from the config file. Their names don't match the real measurement names, so none of them ever match. The file isn't wired in today, but if it ever is, the gate checks nothing.

Each of these ends in a pass. The collector, which is the part that gathers the measurements, even flags an empty run as a failure, but the verdict never looks at that flag.

Why the skip is there

I suspect the skip is on purpose. Two suites share one list of targets, and each suite only makes some of the numbers. Without the skip, each suite would fail on the other one's targets. But "this isn't my number" and "my number disappeared" should get opposite answers.

Missing should mean red

Start from what you expect, not from what showed up. Have each suite say which numbers it owns. If a blocking number is missing, fail and print its name. If nothing was checked, fail. Reject config entries that don't match a real measurement. Print how many targets were expected, checked, missing and failed. Set a minimum number of samples, because a short run tells you little about the slowest cases.

The cheapest place to start is the zero-checks rule, because on its own it would have turned the broken-engine run red.