Flow
Every way a delivery can die, and what it owes
1Out of retries
Goes through the one refund function.
2Endpoint missing at retry
Refunded, not just marked dead.
3Account closed or endpoint off
Refunded at the check before sending.
4Retry past the limit
Settled instead of claimed again.
5Account deletion
One query, then three counts that must match.
6Same path runs twice
Finds the refund and adds nothing.
A webhook in my scheduling app costs the customer a credit, and I take the credit before the request goes out. If the delivery never arrives, the credit should come back. It never did.
How a charge is tied to its delivery
Charging was done right. When a delivery is created, the app saves it and writes the charge to the ledger in one step, so you can't get one without the other. The charge's ID is built from the delivery's ID. That means for any delivery, I know exactly where its charge lives and can check it.
A delivery ends one of two ways. It arrives, and the charge stays. Or it gets dead-lettered, which means the app gives up on it for good. In that case there should be exactly one refund. There never was.
Four places that gave up
Four different places could give up on a delivery: the retry worker after the last try, a retry that found the endpoint gone, the check before sending when the account was closed or the endpoint was off, and account deletion. Each one marked the delivery dead, and most of them erased its contents too. None of them touched the money, and none of them threw an error, because that write is perfectly legal.
The deletion one bugged me most. It ran right before the code that already refunded the account's other paid work.
A single settlement call
Now every path that gives up on a single delivery calls one function, in one transaction. It locks the account's balance first and the delivery second, every time. It checks the delivery is still on the attempt the caller thinks it is, and that it wasn't delivered. It finds the charge and makes sure it's the right account and amount. Refunding is the last step.
It adds the price back to the balance and records a refund entry, but only if no refund entry exists yet.
Because the refund's ID comes from the delivery's ID, running this twice does nothing the second time. Two workers for the same account wait in line on the lock. If anything about the charge looks wrong, it stops and rolls back. I'd rather have a stuck delivery than a made-up refund.
Old attempts can't overwrite new ones
Every claim bumps an attempt counter, and every write about the result has to quote it. A slow worker with an old number can't overwrite a newer attempt.
The web request also moved out of the database transaction. The app checks the account can still send, commits, sends with nothing held open, and records the result after. A slow worker might still send a duplicate, so the signature now covers the delivery ID too. The receiver can spot repeats.
Retries that give up on time
Retries now stop at the attempt limit and go straight to the refund path. Deliveries pointing at a switched-off endpoint are now picked up, which I think is the only way they ever reach the refund.
Deleting an account
Account deletion does the same thing in one big query. Then it counts. The number of deliveries it locked, the number it could refund, and the number it marked dead must all match. If they don't, the deletion step fails. A check that only reads and locks runs first and compares every delivery in the account with the ledger.
Still untested against a real database
The tests use fakes and check the shape of the query. Nothing runs two refunds at once against a real database yet.
And the strictness cuts both ways. One broken ledger entry stops the refund, and in the deletion path that means one bad row can hold up deleting a whole account until someone repairs it.