Flow
Four parts that turn a lost reply into an answer
1ID picked in the browser
Saved before the request goes out, so a reload can find the same job.
2Repeat-safe build
Same ID returns the saved result, and a repeat joins a job that is still running.
3Read-only status call
Done, still running, or not done, and nothing about anyone else's job.
4Reply sorting
A clear yes or no is final. Everything else means check the status.
Clicking the new-order button in the ordering system started a draft build. The dialog said it was building the draft for about a minute and a half. Then it said it failed. But the draft was right there in the order list. The build had worked.
The proxy hung up first
My app sits behind a proxy that won't hold a request open past a set time. The build took longer than that. So the proxy gave up and sent the browser its own HTML error page. My server never heard about it. It finished the draft, saved it, and replied to nobody.
The dialog read every reply as JSON. The error page wasn't JSON, so reading it threw, and the dialog showed "failed." It should have said "not sure yet."
Why it took so long
Most of the time went to pulling stock and sales for every store. Every request went through one shared gate that only let requests start a fixed gap apart. The stores were read one by one. And a long product check ran before any of that started.
What I do with a reply I can't read
I now split replies into two groups. A readable success or a readable "bad request" is a real answer. Everything else is unknown: a server error, a proxy page, a dropped connection. Unknown got its own outcome, so nothing turns it back into "failed."
A readable refusal with a reason is shown as a refusal. Anything else is marked unknown.
The ID and the status call
The browser now picks an ID and saves it before it sends. I added a small status call that only reads. It answers done, still running, or not done. Done comes back with the saved draft. Not done means it's safe to send the same request again, because the server treats a repeat of the same ID as the same job. The status call says "not done" for a job that isn't yours.
If the same request shows up while the first is still running, the server hands back the running job instead of starting a second one. The list of running jobs lives in one server's memory, so it only holds while there's one server.
On an unknown reply, my dialog asks the status call every few seconds for several minutes. If time runs out, it says the draft is still coming and where to look.
Reading stores a few at a time
I now read a few stores at a time and start each read a little early. But I still use the results in the same order, so the totals don't change. The product check runs at the same time, and it still has to pass before anything is saved. My tests check order and that nothing is saved before the product check passes, not speed, but my guess is the wait is about half what it was. The one-server limit on the "still running" list is the part I'd fix next.