Flow
What a store call does now
1Menu first
Read the online menu, same as before.
2Ask for stocked ids
Ask the archive which products have stock at this store. Just the ids.
3Find the gaps
Keep only the ids the menu left out.
4Fetch just those
Look up the rows for those ids and run them through the same filters and vendor checks.
5If the archive fails
Return the old menu answer, exactly.
A product with a few dozen units on a store's floor did not exist to the ordering system.
My catalog service never returned it, so the all-stores sweep called it not found. No row, no reorder.
The cause was the list I read. It's the platform's online menu feed, and a product with online access turned off is left out. It can be on the shelf and selling, and it's still not there. The platform has a master list with every product. So the repair looked easy.
The version that took the service down
The agent wrote the first version and I shipped it. On every store call it also pulled the full master list and the full stock report, and added the hidden products that had stock. The sweep pulled the master list when a seen product was missing. Both big lists sat in a cache for a few minutes.
I never checked how big the master list was. For one store it's well over a hundred megabytes. The sweep walks every store. A few of those lists piled up at once, and the service ran out of memory and died. It came back and died again two minutes later. The ordering system got bad-gateway errors. I reverted it, and the old bug came back. Back to square one.
I should have asked how many copies could be alive at once. A short lifetime doesn't help when the calls come in a burst.
Asking the copy I already had
My service already keeps an archive of the platform's data in a database. So I ask it two small things:
- Which products have stock at this store? Just the ids.
- Give me the rows for exactly these ids.
The new path only adds products. If the archive is missing, not ready, or throws, you get the old answer, exactly. Added rows still go through the same filters and vendor checks as the rest.
Both reads were fast on a live store. The stock read took under a tenth of a second.
When you really want the full list
The full list is still there if a caller asks for it by name. It's read fresh, never cached, and only one runs at a time. A small queue hands the turn on, even when a read fails. That rule is per process, not per fleet.
The archive also runs a little behind. A product stocked since the last sync stays hidden until the next one. And a hidden product with no stock (and, outside the sweep, no sales) won't show up unless you ask for the full list.
Tests check that full reads wait their turn, and that the sweep never grabs the full list when the archive is gone.