Back to Feed
article 4m read

A history chart built before there was any history to put in it

The new history panel in my hub had one tab with real data. Then I recorded on arrival and got lumpy lines, switched to a clock and got a recorder that can't tell a dead sensor from a calm one, and found a read side that can cut off the newest rows once the table grows.

Article DepthPractice

Flow

Checks I'd run on my history panel

One check for each problem in this post.

  1. 1A fast and a slow sensor side by side

    After the clock change, both should end up with the same number of rows.

  2. 2Unplug one sensor

    The chart should show a gap or a stale mark, not a calm flat line.

  3. 3Fill a window past the cap

    The right edge of the chart should still be now.

  4. 4Compare a card's max with the raw max

    If they differ, the card is showing the highest average.

  5. 5Set one device to a different unit

    See whether the history chart agrees with the device screen.

The new history panel opened on the energy tab for a reason. Energy was the only metric that was already being saved. Every other sensor had a tab and nothing behind it. The next commit was there to fix that.

None of this came with tests, so what follows is what I read in the code, not what I saw run.

Memory for now, a table for later

Devices send their state over MQTT. The hub keeps the latest message from each device in memory, and the dashboard reads that. It knows the present. It keeps no past. History goes into a small SQL table of time, device, type and a blob of values.

Saving on arrival

The first try saved a row when a message came in, at most once per device per interval, so a chatty plug couldn't flood the table.

The commit that replaced it says what went wrong: slow sensors, like temperature, showed up at random intervals. A limit like that stops a device from being saved too often. It does nothing when a device speaks too rarely. The chatty plug got a row every interval. The slow sensor got one whenever it spoke. Each device's firmware was setting the spacing of my chart.

Saving on a clock

The second try is a background loop. It wakes up on a fixed interval, copies the cache and saves one row per device. Now every device is on the same beat.

It has three weak spots. The loop sleeps first and works second, so the beat slowly drifts. If the first call to connect to the broker throws, the loop never starts, and nothing tells the panel. And it never checks how old a value is. A dead sensor keeps its last message, so the loop saves that same number on every tick, and the chart shows a calm flat line. What I want is a seen-at time on each cache entry, plus a stale marker once a value gets too old. I haven't built it yet.

Where the newest rows fall off

The chart's query takes rows in the time window oldest first, stops at a fixed cap, and only then picks out the metric you asked for.

Everything in the window counts against the cap, whatever metric it belongs to.

Other metrics use up the cap. And if the window holds more rows than the cap, the newest rows are the ones that get dropped. That's the right edge of the chart, the part people read. I haven't seen it happen. But saving every device on every tick is exactly what grows the row count. Filter by metric in the query, count from the newest end, and say when you cut something off.

The tab badges have the same bug in small: they treat the oldest row of the day as the latest. Live values from the cache cover it up for most tabs.

What a number on a card means

Readings go into time buckets and get averaged, first per device, then across devices. That's fine for temperature. For power it's wrong: an average per plug is not the total. The cards for min and max come from that averaged line, so short spikes vanish. The energy tab keeps the first reading in each bucket instead, so two tabs use two different rules. If you build one of these, decide for every number on screen whether it's an average, a max, a sum or the last reading, and label it that way.

A lazy chart and a unit setting it ignores

At first the chart only loaded on page open or when I switched tabs. A later commit gave it its own slower refresh, apart from the badges.

Temperature moved from Celsius to Fahrenheit, converted in one place so every number agrees. The same day, each device's unit setting became editable, and the history code ignores it. That works if every sensor sends Celsius, and I think mine do.

If I started over, I'd put the age check in before drawing the first line.

Keep reading

Pick up a connected idea or branch into a nearby one.

5 paths forward