Flow
Making a hook budget real
1Create signal
One abort signal per invocation, sized to the inline budget.
2Combine
Merge it with each request deadline so either one cancels the call.
3Gate batches
Check the signal before every new batch; started work is still awaited.
4Leave pending
On abort write no terminal status and no partial results.
5Resume later
An explicit command finishes pending runs with the large budget.
The first version of my shadow model could leave someone stuck on the order screen for a couple of minutes. They were waiting on an answer they would never see. The order was safe, but the wait was real.
The ordering system has a rules engine that decides how many of each product each store gets. I trust it. I wanted to know if a language model could do the same job, without letting it touch a real order.
So the agent built a shadow. After a draft order is saved, a hook asks the model what it would have ordered for every row, even rows the engine skipped. The answers go into their own tables and nowhere else. Not the order, not the paperwork, not the supplier. A few separate commands read them, and one compares the model to the engine. A switch turns it off.
How each row's choices get built
The model never writes a number. A free number could hide a math slip or a forgotten case size, and then bad judgment and bad arithmetic look the same. Instead, the code builds a short menu for each row: zero, the engine's pick, and whole-case steps up to and including the first one that's clearly too much. That last option is how the model can show it would overbuy. The menu is labeled by unit count only, and it never says which option was the engine's. If the answer doesn't match the menu, save it as invalid.
What the model gets to see
The first cut used raw IDs as names and gave the model almost no facts. Review fixed that the same day. Now each row has real names and plain facts: units sold, days in stock, sales speed, stock on hand and stock on the way. Leave out anything only the engine could work out, like its target. That's just the answer in disguise. Tell the model that zero stock means sold out now, and that product names are data, not instructions.
Where the answers live
Keep the shadow tables unlinked from the real order, so nothing in the order screens can lean on them. Save failures as a short code, never the full error, since errors can hold secrets.
A timer that only stopped me waiting
The hook ran while the person waited, and it was bounded by the long timer meant for the command line. Worse, the timer only stopped the waiting. The loop kept starting new model calls in the background. I missed both. Now:
- the hook gets a few seconds, not minutes
- one cancel signal goes all the way down to each model call
- no new batch starts once it fires
- an unfinished run stays pending, not failed, and a later command finishes it
The hook creates one signal that expires after a few seconds and passes it all the way down.
Leaving the run pending keeps a slow provider out of the failures in the comparison. Tests check each step.
Before the numbers mean anything
I don't have results yet. The plan is to collect pairs of answers on real orders and have the buyers look at every disagreement.
Rows where the model gave no usable answer should count as missing, not as a disagreement. I also suspect an option's position in the list might matter, and nothing here checks that.