Flow
Signs the column is wrong
What I would look at once queries read the new column.
1Empty column, full map
Rows where the column is empty but the map has the id point to a missing copy or fill.
2Old lookup versus new column
Per-tenant counts should match over the same range.
3Older date ranges
Charts that go blank for older dates are the visible symptom.
4Live column rule
The rule each database actually holds, since a skip-by-name statement can leave an old one.
Flow
Three sources for one column
What each version needed in order to be right.
1Computed from the row
Needs only the map that every row already carries.
2Default over a copy
Needs the collector copy step, deployed in order, for every row ever written.
3Default plus a fill pass
Needs the fill to run to the end on every database.
4Speed claim
Came from general benchmarks. Not measured on my data.
The column all my dashboards filter on changed where it gets its value three times in a few hours. The first version was the right one.
My telemetry backend stores events in ClickHouse. Each event carries ids that say whose it is: org, team, user, and key. They sat in a generic map, so every tenant filter had to dig through that map on every row. A map value can't be indexed well. So the filter that runs on every dashboard was the slowest kind.
I had an agent move the ids into real columns with indexes and point the queries at them. Good plan.
Three sources in one day
The first version made the column derived.
The column is just the tenant entry from the map, worked out by the database for each row.
Nothing upstream had to change, and old rows have the map too. I believe the database fills old rows from that same rule when it reads them, though I haven't proven it.
The second version had the collector copy each id into a second map, and the column read the copy. It was pitched as a storage win. But now the column depended on the collector, and three parts had to ship in order. The comments promised a fallback to the original map, and there wasn't one. Old rows never got a copy, so they probably read as empty.
Then the collector wouldn't start. One copy rule used a path its config language doesn't allow, and a bad config stops the whole collector. As I read it, that meant no events got in for anyone. The collector can check its own config first, and nothing in the change ran that check.
The agent also folded the columns into an older migration file. Each statement skips a column that already exists, by name. So a database that ran an older version keeps its old rule and still says it succeeded.
The third version went back to computing from the original map. A separate human commit added a fill pass for empty rows instead. They're on different branches, and I can't tell from the commits which one is live.
I also never measured the speedup. The agent's numbers came from general benchmarks, not my data. The test I trust is plain: count rows per tenant with the old lookup and with the new column, over old data too, and check the live column rule on every database instead of the migration file.
An empty id matches no tenant, so nobody sees someone else's data. If old rows are empty, people just see blank charts for older dates.