Flow
Picking checks for a feed
1Slow catalog?
Totals should stay steady, so keep all four checks.
2Stock that grows in bursts?
Keep only the drop check and the shared-ID check, since missing rows are what close history.
3Live list of mismatches?
Shrinking is the goal, so skip the comparison and allow an empty list.
4Rejected anyway?
Store it as evidence and try again after a short, capped delay instead of waiting hours.
One guard in my archive would look at a restock and call it bad data. A delivery comes in, the package list gets longer and the numbers get bigger, and the guard says no.
Missing rows close history
The archive copies data from the point-of-sale system the stores use. Many feeds send the full list every time. If a record is missing from a full list, the archive marks it as gone. That is how it learns about deletions. It is also the risk: one cut-off response could wipe out a lot of real history.
So a guard compares each new list with the last good one. It checked for a big drop in rows, a big jump in rows, too few IDs in common, and a jump in total quantity. If a list still looked wrong after a few tries, the archive stored it as evidence but did not use it.
The guard used the same checks on every feed. It assumed every healthy feed changes slowly.
A restock breaks that
Product catalogs do change slowly. Package feeds do not. A delivery adds many rows with big quantities, and all the old rows stay. The jump checks reject a correct list. The reconciliation feed lists current mismatches, so a shrinking list is good news, and the drop check rejected that too.
It gets worse. A rejected list never becomes the new reference. So the next try compares with the same old list and fails again. A real change stays stuck until someone accepts a new baseline by hand.
Catalogs, packages and mismatches
Now each feed says what kind of data it is:
- Catalogs keep all four checks.
- Package and inventory feeds keep only the drop check and the shared-ID check.
- The reconciliation feed and the live stock feed skip the comparison and may be empty. The reconciliation feed also stopped counting toward overall health.
The two checks that stay on every compared feed are the ones that guard against missing rows. Missing rows are what close history. A bigger list, or bigger numbers, closes nothing.
Retrying sooner
A rejected feed used to wait for its next normal run, which could be hours away. Now it tries again after a short, capped delay. Retries still go through the rate limiter, and bad data is still refused. One side effect: a feed that stays broken will pile up more stored copies of its bad responses. I haven't picked a limit yet for how much of it to keep.