Back to Feed
article 3m read

Three test runs could collide on one workstation while I was busy tuning compilers

Compilers kept getting reshuffled inside one validation run to protect their deadlines, but the bigger risk was contention between local, CI and release runs that could all start on the same workstation. A loopback port now gives each run the machine to itself, the kernel frees it on a crash, and queue time is recorded apart from run time.

Article DepthPractice

Flow

What the lock needed and how a port covers it

  1. 1Callers in different distributions

    They share a network but not files, so a lock file is out. Every caller sees the same loopback port.

  2. 2A run crashes

    The kernel frees the port when the holder dies, so the lock never stays stuck.

  3. 3No extra service

    Binding a port needs nothing else running, so there is no lock service to look after.

  4. 4Port already taken

    The bind fails with "address in use", and that simply means wait.

Compiler scheduling in my test suite kept getting tuned, and I started to suspect the bigger risk wasn't in the suite at all.

The suite is one command that checks the ordering system: unit tests, a throwaway Postgres, lint, type checks and builds. Each task has a hard deadline, and so does the whole run. A release only gets signed if the run passes inside those limits, so a late run counts as a failure.

Tuning the wrong layer

The type check over the whole app was the task I worried about, so I worked on the scheduler. First the heavy tasks had to take turns. Then the big type check got the entire CPU pool to itself while it ran.

Each change was meant to protect one run from its own tasks. But three different callers can start the suite on the same workstation: me testing local edits, the CI job after a push, and the signed release run. None of them knew about the others, so if two ran at once they would split the machine between them. A timeout caused by a neighbor looks exactly like a timeout caused by slow code.

Why a loopback port

My callers run as different users, inside different Linux distributions under WSL. Those share a network but not their files, which rules out a lock file. A crashed run must not leave the lock stuck, and I didn't want a separate lock service to look after.

A listening socket on a fixed loopback port handles all of that. Only one process can bind it, and when that process dies the kernel frees the port on its own. Anyone else who tries gets "address in use", and that error simply means wait.

The waiting loop

Try to bind. If the port is busy, print one line, sleep briefly, and try again until the queue budget is spent. Let Ctrl-C end the wait cleanly, and treat any other bind error as a real error. After you win, check one more time:

if (won && (cancelled || waited > budget)) { release(); throw new Error('too late') }

That stops a run from grabbing the port after it has already run out of time or been cancelled. Release the port in a finally block. Commands that only print a plan should skip the lock completely.

Deadlines stay put

Raising the deadlines would have hidden the waiting, and real slowdowns along with it, so I kept every deadline the same. Instead the receipt now stores queue time next to run time, and the checker rejects a receipt with no queue time or with too much. Only the outer CI and release timeouts grew, by exactly the size of the queue budget, because the outer limit should always fire last. With the whole workstation owned by one run, the type check's full-pool reservation went, and later that day a smaller one came back to keep it apart from the big unit-test worker pool.

The child process that gets killed

My tests show that a second run waits until the first lets go, that a short budget fails with a clear message, and that a cancelled wait fails without freeing someone else's slot. The best one starts a real child process that holds the port, kills it outright, and then shows the port comes back.

Still open: another program sitting on the same port would look exactly like a busy slot, and a run would wait out its whole budget for nothing.

Keep reading

Pick up a connected idea or branch into a nearby one.

5 paths forward
Related idea

Device limit counts first and saves later, and the test never sends two at once

The licensing code I shipped checks how many devices a key already has, then inserts the new one in a separate step, so two activations racing for the last slot can both get in. The test activates devices one at a time and can't see it, a device racing itself gets a server error, and a key-only check passes for keys never bound anywhere. I found this by reading, not by running anything concurrent.

via #concurrency · #testing