Back to Feed
article 3m read

The import flag my docs call critical just drops every matching view and rebuilds it empty

A post-import step meant to backfill analytics views drops and replays each view's stored definition, which restores the trigger but not the rows, touches every org's views, and swallows failures, all to fix a gap nobody ever counted.

Article DepthPractice

Flow

What the counts tell you to do

  1. 1Counts match

    Stop. The views already saw the imported rows.

  2. 2View short by the imported rows

    Insert the missing rows with the view's own query, up to a cutoff time.

  3. 3View short some other way

    The view filters rows out. Look at its query, not the data.

  4. 4After any backfill

    Count again and treat a mismatch as a failed run.

Its docs call the new flag on the import tool critical. All it does is drop each analytics view and make the same view again.

I'm going off the diff and the docs. I didn't run it, and I don't have counts.

One org, copied into shared tables

The tool copies one org's data from one database to another and changes the org and user IDs as it goes. It puts the rows in the raw tables. The docs say the dashboards depend on views built on top of those tables.

The new flag is off unless you pass it. When it's on, the tool finds the views, reads how each one was defined, drops it, and runs that definition again. A comment says this pulls in old rows. The docs say to always use it.

Why the problem may not exist

The docs say views only pick up rows inserted after the view exists. That's how I understand them too. But an import is a fresh insert, and the views are there when it runs. So the imported rows should already go through them.

Rows only go missing if a view was made after the data, was gone during the import, or filters the rows out. The change never counts anything to find out. There's a counting helper in the same file, and the step doesn't use it.

Why the step doesn't help

It never adds the option that copies old rows in. It just replays the saved definition, which I don't think keeps that option. So the new view starts empty. If the view kept its own rows, the drop deleted them. If it wrote into a separate table, those rows stay, and nothing new is added. Best case, nothing changes.

It also hits every view, not just the ones tied to the table that got data, and it does it once per table. One org's import rebuilds views every org uses. While a view is dropped, new rows from anyone skip it. If making it again fails, the tool logs it and keeps going, and still says it finished. There are no tests.

Counting before touching anything

First, count. Count the org's rows in the raw tables and in each view, before and after the import. If they match, stop. If a view is short, fill the gap with an insert that runs the view's own query over the missing rows, up to a cutoff time:

INSERT INTO hits_store SELECT team_id, count() FROM raw_events
WHERE team_id = 'imported' AND event_time < '2000-01-01' GROUP BY team_id;

Leave the views in place, and count again after. For a fresh setup, make the views before you load data. Until someone runs those counts, I'd leave the flag off, whatever the docs say.

Keep reading

Pick up a connected idea or branch into a nearby one.

5 paths forward