Back to Feed
article 3m read

Inventory archive had no way to believe a real stock drop

A guard that rejects suddenly shrunken stock snapshots had no way to accept a real drop, so I made a failing feed ask its sibling feeds to agree on size and items before it gives up.

Article DepthPractice

Flow

How a failed list finds a witness

  1. 1List fails the check

    Much shorter than the baseline, or mostly different items.

  2. 2Recent good sibling?

    Another feed for the same store that pulled successfully not long ago, judged by its own schedule.

  3. 3Recently rejected sibling?

    If not, load a sibling list rejected in the last couple of hours and re-check it.

  4. 4Close in size and items?

    Sizes within about a quarter, and most of the items match.

  5. 5Accept and log it

    Record why it almost failed and which sibling vouched for it.

  6. 6No witness, keep the no

    The rejection stands, and I can still accept a baseline by hand.

Stuck for good was the risk of a guard in my inventory archive, and a store doing something normal was enough to trip it.

The archive pulls each store's stock lists on a schedule and keeps every one. Each new list is checked against the last one I accepted, the baseline. If the new list is less than about half the size, or has mostly different items, it gets rejected. The raw copy is saved as evidence, a retry is set for a few minutes later, and the baseline stays the same. I want it that way. A wrong list saved as truth damages my history and nobody notices. A good list rejected is noisy, and noisy is easy to spot.

When the drop is real

Now say the drop is real. A store clears out a lot of product, and every stock feed for that store shrinks at once. Each one is checked against its own old, bigger baseline, and each one fails. A rejected list can never become the baseline, so the next pull fails too, and so does every retry. The only way out was me accepting a new baseline by hand, one feed at a time.

Making the rule looser would make every feed easier to fool. So I left the rule alone and gave it a second opinion.

Grouping the feeds

A few of my stock feeds cover the same store's stock, so they're grouped now. When one fails, it holds the rejection and asks the others first.

If the check fails and no sibling backs the snapshot up, it is rejected. Otherwise it goes through.

A sibling counts as a witness only if it agrees on two things. The sizes are close, within about a quarter. And most of the items match. You need both. Two broken lists can be the same size by chance, and a short list of the right items is just what a cut-off response looks like.

The witness can come from two places. Best is a sibling that pulled successfully not long ago, judged by its own schedule. If there isn't one, the archive uses a sibling's list that was itself rejected in the last couple of hours, loads it again, re-checks it, and compares. That second path is what gets a store unstuck. The first round of lists is rejected, and on the next round they can vouch for each other.

Lists that must stay rejected

My tests cover four cases: a fresh sibling agrees, a recently rejected sibling agrees, nobody agrees, and a sibling is the right size but has different items. The last two have to stay rejected, and they do. Every accept that needed a witness gets logged with the reason it almost failed and which sibling vouched for it, so I can explain it later. I can still accept a baseline by hand when nobody backs a list up.

What the tests can't tell me is how often this gets it wrong in real life. I haven't measured that.

Keep reading

Pick up a connected idea or branch into a nearby one.

5 paths forward