Flow
What the backtest changed
1Old formula guessed low on slow sellers
Borrow from the same product at other stores when history is thin.
2Ranges too narrow on fast sellers
Add day-to-day spread so the range catches the truth as often as it claims.
3Trend tweak
No gain, so it is off.
4Scaling by other stores
Made stock-out guesses worse, so it is a clue only.
5Halving the guess
Came out low by about half, so it is never the default.
A backtest I built hides sales I already know and grades each demand estimate on how well it guesses them back.
The reason is empty shelves. When the shelf is empty, sales are zero, and a plain average of daily sales would read that zero as nobody wanting it. The risk is a loop: a smaller order, another empty shelf, more zeros. If that loop were running, each turn of it would look like proof.
Guessing the empty days is easy. Checking the guess is hard, because nobody wrote down the real answer. So I checked it somewhere the answer was known.
One daily table
The agent and I made a table with one row per store, product and day. It says how the shelf looked at opening, what sold, the price, any promotion, and whether the store even carries the product. The model and the backtest both use it, and they share one definition of a clean day: a full shelf, a product the store carries, no promotion or markdown, no data problem.
That table paid off right away. About two in three store-and-product days were products the store had never carried. They are not stock-outs, and they shouldn't count as lost sales.
One weak spot: a shelf that is full at opening and sells out by afternoon still counts as clean.
Grading on stretches with a known answer
The backtest finds short stretches where every day was clean, so the true sales are known. It hides them and asks each method to guess using only earlier days, then compares. A fixed seed means a rerun picks the same stretches. It scores three things: how far off, which way it leans, and how often the truth landed inside the claimed range. The lineup is my old formula plus versions of the new model with one piece changed, so each piece has to prove itself.
Less lean, wider ranges, same error
My old formula guessed low on slow sellers by close to a fifth. The new model borrows from the same product at other stores when history is thin, and that mostly fixed the lean. Its ranges for fast sellers were too narrow: they claimed four in five and caught about two in three. Adding extra day-to-day spread fixed that. A trend tweak didn't help, so it's off. Overall error didn't improve, and that's fine. A few days of sales are mostly luck.
The no-evidence case
I didn't catch this one; it was pointed out to me. With no history and nothing to borrow, the model's tiny starting value read as one unit a day. Now it returns zero and says there was no evidence. A bigger starting value was tried and pushed slow sellers up, so it was dropped.
Real stock-outs and order replays
For each empty stretch, the system offers a few guesses and a language model picks one, with a formula as the backup. The test chose that list. The product's own pace was close. Scaling it by other stores made it worse, so that's shown as a clue only. Halving guessed low by about half. All of those numbers were measured on products that didn't run out.
To compare two ordering plans, each order is replayed against what really sold, with the model filling in only the empty days. The first try covered one order that wasn't finished yet, so it proved nothing.