bar chart
Average of averages versus the real average
Made-up example, not measured data. One summary holds a single 1,000 ms request; the other holds 9,999 requests averaging 100 ms.
Each summary gets one vote, so the lone slow request pulls hard.
Every request gets one vote.
Flow
What I'd fix, panel by panel
Read from the diff, not from live charts.
1Latency panel
Use the band counts for p95, or rename it to average, fastest and slowest.
2Cost and token totals
Pick metrics from a fixed list of names, not a word in the name.
3Running totals
Check whether a row is a running total or a change before adding it.
4Token total
Mark the switch to counting cache tokens.
5Slow lane
Count the same kinds of events on both sides before comparing.
Dashboards in Grafana for my AI agents got rebuilt in one commit. When I read the diff back, one panel stood out. Its title said p50, p95 and p99 latency. Its query never works out a percentile. Nothing would ever warn me about that.
I'm going on the diff alone here. I didn't check live charts, so this is about what each query does, not what a graph showed.
Fast lane, slow lane, and a third dashboard
My main dashboards used to dig numbers out of raw log text on every query. I moved them onto a view that stores those numbers ready to use. That's the fast lane. I kept one dashboard on the raw logs to compare against. That's the slow lane. Then I added a dashboard over my metric tables, and the latency panel lives there.
Why that panel can't be showing p95
A histogram row is not one request. It's a summary of a stretch of time: how many requests, their total time, the fastest, the slowest, and how many fell in each time band. The panel turns each row into an average, then averages those. A row with one request gets the same say as a row with thousands. An uneven pair of rows shows it.
One row holds a single slow request and another holds ten thousand quick ones. The chart shows around five hundred fifty while the real average is about a hundred.
And neither is a percentile. For p95 you need the band counts, added up across rows, and the band where the 95 percent mark falls. The query never reads the bands, so it can't say a thing about the slow requests. Those are the reason p99 is on a chart.
What bugs me is that my tool dashboard, in the same commit, gets this right. It keeps one row per tool call, so a real p95 is easy. The metrics panel only has summaries and still wears the label.
Smaller labels that overclaim
Once I was looking for labels that promise too much, I found more.
- Cost and token panels add up every metric with a certain word in its name, so a new metric with that word joins the total. Use a fixed list of names.
- Some metric rows are running totals, not changes. Adding running totals counts old work again. No panel checks which kind it has, and the diff can't tell me what my senders send.
- The main token total used to be input plus output. Now it also counts cache tokens. I changed the description but didn't mark the switch, so anyone comparing with the old number would see growth that isn't real work.
- A panel that claims to show busy hours of the day is really a plain timeline.
- My test data script samples each table at random on its own, so its logs and metrics don't describe the same requests. It also puts the fake user where my cost-by-user panel doesn't look.
Where the latency panel goes from here
The slow lane only works as a check once both sides count the same kinds of events, and today they don't. So it can't catch the latency panel for me yet.
I see three paths. Rename it to average, fastest and slowest, and work out the average as total time over total count. Merge the band counts when the query runs. Or store merged bands ahead of time, the way the fast lane already stores requests. I want the slow tail, so I'd go with the bands.