Back to Feed
article 3m read

A lock on settings saves gave stale admins a place to wait

Making settings saves queue on one workspace lock opened a gap: the permission check ran before the wait, so a signed-out, demoted or expired admin could still write. Now authority is rechecked under the locks, with second-factor proof read outside them and validated again at the end.

Article DepthPractice

Flow

Three steps for one settings save

  1. 1Step 1: database check

    Check the user, session and role without changing anything. Stop early if any of them is already bad.

  2. 2Step 2: read the proof

    Without holding a lock, read the second sign-in proof and how long it has left. Give up if the cache doesn't answer in time.

  3. 3Step 3: lock and check again

    Take the locks in the usual order, run the checks again, and make sure the proof still belongs to this session and hasn't run out.

  4. 4Write

    Save the settings only after every check passed, and write the audit record if it is a security change.

Waiting in line was my new rule for settings saves, and that gave stale admin sessions a place to stand.

Two parts of my app could save workspace settings, and they didn't share a lock. A login could also read the security policy while a save was changing it. So all of them now take the same lock on the workspace. Saves wait their turn, and logins read the policy only once they hold the lock. That part was right.

But the notes for that change already named the catch. Checking who you are at the door is not the same as checking again after you wait.

What can change while a save waits

A guard checks the user's token and role when the request comes in. Then the save waits for the lock, for as long as the save ahead of it takes. Then it writes, trusting a check made before the wait.

During that wait, a lot can change. The admin can sign out somewhere else, get suspended, or lose the role. The workspace can be shut off. Someone can shorten the session timeout, so this session is now too old. The save still goes through.

The recheck under the lock

Now the check happens when the locks are held. The save locks the workspace, then the user, then the exact session from the request, then reads the current roles. The order is always the same, so two saves can't block each other forever.

Then it checks again. Is the account still active? Is this session still live? Does a current role grant the right to edit settings, not just a claim on the token? Is the workspace still open? Is the session still inside its time limit under the policy as it is now? If any answer is no, the save stops before it writes anything.

The time limit is the earlier of the session's stored end and its start plus the current timeout. A save that asks for a longer timeout can't rescue an admin who is already too old.

Second-step proof in the cache

Proof of a second sign-in step lives in a fast cache, not the database. Reading it while holding locks would make every waiting save as slow as that cache. So the save checks the database first, then reads the proof and how long it has left without holding any lock, then takes the locks, checks again, and makes sure the proof is still good. If the cache is down, the save fails and can be retried. It never passes by default.

The same mistake showed up in login, refresh and a few account security flows. They all lock first now.

Fake transactions, real gaps

The tests pause a save at a lock, change something, and check that the save is refused with nothing written. They use fake transactions, so they don't prove how a real database blocks. Older builds that skip the lock can still overwrite settings. Until every one of them is gone, the fix only covers the writers that know about it.

Keep reading

Pick up a connected idea or branch into a nearby one.

5 paths forward
Continue the thread

The draft button checked for duplicates, then left a gap wide enough for a second one

The button that saves a delivery as a draft in the stores' inventory system checked first and wrote its record only after a slow outside call, so a second click could slip through. Now it claims the delivery in the database first, lets a partial unique index refuse the second claim, fixes a payload hash that couldn't see nested fields, and adds a narrow admin cancel for claims that get stuck.

via #concurrency · #databases
Related idea

Device limit counts first and saves later, and the test never sends two at once

The licensing code I shipped checks how many devices a key already has, then inserts the new one in a separate step, so two activations racing for the last slot can both get in. The test activates devices one at a time and can't see it, a device racing itself gets a server error, and a key-only check passes for keys never bound anywhere. I found this by reading, not by running anything concurrent.

via #concurrency · #authorization