Back to Feed
article 3m read

One store's bad day was taking down every vendor report I built

A missing store used to kill the whole supplier report, a single request line let one slow read stall everyone, and one store's overdue data made the gateway call itself unready. Now each store fails on its own, reports say exactly what they covered, and partial ones can't be emailed.

Article DepthPractice

Flow

What changed, in the order a request meets it

  1. 1Request line

    The body is read inside the slot on the report clock; a few requests overlap while starts stay spaced.

  2. 2Turns and deadlines

    Each report run gets its own queue and its own deadline, and a user leaving drops the waiting request.

  3. 3Readiness

    Fails only when something every request needs is down; store problems go to a separate status response.

  4. 4Per-store resolution

    Live data, then a fresh and complete saved copy for that store, then marked missing.

  5. 5Send rule

    The preview shows covered stores and the oldest data time; email waits until every store is there and fresh.

A single rule applied to the vendor reports: every store reads cleanly, or nobody gets a report. If one store's read failed, the whole report threw an error. One store's problem took down the preview for every other store.

The rule made some sense. These reports get emailed to suppliers, and a report that drops a store without saying so is worse than none. But there was a middle option, and I'd missed it in three places.

The line every request stood in

Reports pull each store's data through a gateway in front of the point-of-sale system. That system limits how often requests can start. My side kept one line with one request at a time.

But the limit is about starts, not about finishing one before the next. One slow store held up everyone. And the line let go of a request as soon as the headers came back. The body was read later, with no timer on it at all. A stalled body could hang a report while the line thought the work was done.

Now the body is read inside the slot, on the report's clock. A few requests may run at once, and starts are still spaced the same way. Each report run gets its own queue, and the scheduler takes turns between them. Each kind of report has its own deadline. If the deadline passes or the user leaves, the waiting request is dropped before it's sent.

The page already ignored old previews when you switched vendors. It just never told the server to stop. Now it does.

A readiness check that failed for everybody

The gateway has a ready-or-not check that my deploy scripts wait on. It said "not ready" if one store's data capture was overdue, if the background worker was stale, or if storage was low. So one lagging store made the whole service look broken.

Now each problem says what it affects and whether it blocks everything. Readiness only fails when something every request needs is down, which after a later commit meant just the shared rate limiter. Store problems show up in a separate authenticated status response, by store, with a code.

Reports that admit what they're missing

Each store now gets resolved on its own: live data, or a saved copy if it's fresh and complete for that store, or marked missing. The preview shows whatever stores it has. It says how many stores are covered, names the missing ones, and shows the oldest time any data came from. Email is blocked unless every store is there and fresh.

The credit report needs both stock and cost history for a store to count. When a product comes partly from a saved copy, the credit is recalculated on the combined total.

The timestamp can look worse than the data

The as-of time is the oldest source, so a mostly live report can show an older timestamp than most of its data deserves. Running a few requests at once also puts more load on the point-of-sale system.

Keep reading

Pick up a connected idea or branch into a nearby one.

5 paths forward