bar chart
Which calls go first
A higher bar is served first. Inside one class, the call that showed up first goes first.
A person is waiting on it.
Something I started by hand.
Regular snapshots on a timer.
Follow-up after a snapshot worked.
Old history. Gets whatever is left.
Two of my programs were using the same rate limit, and neither one knew the other was there.
One is the web service in my gateway. The other is a worker that runs in the back and pulls data from the same vendor. The vendor lets me make a set number of calls each minute, and I read that as one limit for both. The worker had its own caps. The web side had none. Nothing watched the total.
The first patches
Earlier that day I made the worker less pushy. A batch never hits the same store or the same source twice at once. When the vendor says slow down, the client waits as long as the vendor asks, or a short floor if it doesn't say. Data that is the same for every store gets fetched once. All good, but none of it could see the web side.
One line for everyone
So now every call waits in one line, retries too. A test checks that each try gets its turn before it goes out. Each call has a class. Calls a person is waiting on go first, and backfill of old history goes last. Ties go to whoever showed up first. A status view shows how much of the minute is used and how many calls wait in each class.
The line lives in the Postgres database both programs already use. It holds the next time anyone may send, the list of waiting calls, and a log of past sends. Each waiting call has a lease its program keeps renewing. If a program dies, its leases run out and its calls get cleared. A test with two separate copies of the gate shows an urgent call jumping ahead of old backfill. If the database is down, the gate turns calls away with an error that says calls are paused. Guessing is what I was trying to stop.
The timing bug
Later that day I went back to the spacing, because it was not even. Each next slot was timed from when the last send really went out, and timers run a little late. So every gap ran long, and the lost slots never came back. Now small slips are absorbed.
A send that's only slightly late keeps the original beat, and the schedule resets from the actual send time only when it has slipped more than a whole gap.
That can let two sends go out close together. So the gate also counts the log. If the last minute is full, it waits. A test queues one call too many and shows that one waits its turn.
Startup also refuses a database pool too small to leave room for heartbeats during a big batch.
The test for the minute count uses fake timers, and fake timers are never late, so it never sees the exact problem I fixed. Old backfill can wait a long time when busy work keeps coming, and only the status view would show it. The outage path has no test I saw, so for now I am trusting the code on that one.