Back to All Posts

#reliability

tuckerl.ee

Building thoughtful software with AI.Notes from the systems behind the work.

Also new

article

Zero ordered for an empty shelf, and the planner never said why

A stock ceiling and my empty-shelf guarantee collided on a single supplier case, and the planner settled it by dropping the product with no reason. I ranked the rules, let one store go over on the record, and threw out the agent's first odd-case version because it pushed a stocked store over its own cap.

article

Failed uploads the cleanup couldn't find, so the replacement I built mostly refuses to delete things

A failed upload would have left its partial archive behind because the cleanup only learned the file's name from a return value. I fixed that, then built a scheduled cleanup that decides what it must keep before it deletes anything, checks every target on its own, uses a lock that dies with its owner, and retries an upload exactly once, only when it runs out of disk space or quota.

article

Fake cable channels started every show at the opening credits

Tuning into one of my home-made channels always played the first item from the top. Now a channel is a half-hour clock: the hub works out what would be on and how far in, starts there, then reads back where the player really landed and seeks once if it didn't listen. Long movies are still stuck in their first half hour.

article

Reorder approvals that lived in memory while the job outlived the request

The slowest assistant job, building a reorder, ran inside the request on an in-memory approval token, so a timeout or restart left nobody knowing whether a draft order existed. Now apply writes a synced journal entry before it answers, workers hold fenced leases, an old approval can never restart a job, and an expired lease reads as interrupted rather than failed.

Reviewed purchasing plans were stamped by the database clock and checked against the app server's clock, so a few seconds of skew could make a brand-new plan look like it hadn't been made yet. The authority checks now read the database clock once, in the same transaction (right under the locks, for approval), with no fallback.

article

Backups that only got a second copy on Sundays

A nightly copy of this box's backup repositories now goes to my NAS and proves where it's writing before it writes, and I also fixed a checksum that would always fail for the wrong reason and an install guard that could never fire.

article

Releases never owned the browser the worker depended on

The one worker that drives a headless browser only learned at job time whether a browser existed, because the browser lived in a user cache instead of the release. I sealed it into the release with a manifest, made health refuse promotion without it, and then caught my own guard looking for it in the wrong folder.

article

Test suite deleted a store it never created

A browser test accepted an unattended confirmation and cascaded a delete through a shared seed store in my local test database. Now every destructive test request has to prove the database is disposable and that its run owns the exact record it touches.

article

A supplier setup lease that trusted a name tag and the worker's own clock

The lease guarding my long supplier setup job only checked the owner's name, judged expiry by each worker's clock, and never renewed during slow phases. Now every claim bumps a fencing token, every write checks it against the database's clock, the phase-advancing writes also check the phase they expect to leave, and a heartbeat keeps slow work alive.

article

Due-date check called correct handoffs failures during daylight saving

The nightly audit job read back each task's due date and could flag correct handoffs as failures, because the task tool fills a date-only due date with a local morning hour whose UTC hour shifts with daylight saving. The check now compares by day when only a day was sent, and a tightly gated command can finish handoffs that were wrongly marked failed.