Flow
Swapping root for a one-VM token
1Spot the caller
The drift repair ran the updater with no human
2Name the job
Run a few commands in one VM
3Make the token
One privilege, one VM, user and token both granted
4Pin, then send
Check the fingerprint before adding the token
5Guard the script
Flag SSH and the old root address
Any deploy of the ordering system that saw drift on its release gatekeeper would grab the root SSH key for my Proxmox host and use it. That key controls every VM I have.
How the repair reached for root
Before each deploy, the client checks the gatekeeper programs on one of my VMs against the repo. If they differ, it fixes them. That repair ran an updater that logged into the Proxmox host as root and pushed a package into the VM.
The updater's docs said a human should run it on purpose. The client ran it by itself on drift, after finding the root key on disk. Probably the updater was written for me, and the drift check picked it up later because the key was easy to find.
Too big a key
The job was to run a few commands in one VM. The key could do anything to any VM, and to the host. A bad drift check or a compromised build step would have had all of that.
Making the repair manual again wouldn't have helped much. The key would still be on disk, and a script would end up holding it again.
One permission on one VM
The agent wrote the replacement and I went through the diff. A one-time script, run by a human with the root key, sets up three things in Proxmox:
- a role with one privilege: running commands through the guest agent
- a user just for this VM
- a token under that user, with the role granted to both the user and the token, on this VM only
The commit says the scope was verified against the live host in both directions. The diff doesn't include that output, so I'm trusting the commit for now.
Now the client can use the token on its own again. It reads it from a locked-down file and passes it in an environment variable. If the token is missing, it stops and says how to make one. It never falls back to root.
Checking the certificate before the token goes out
Proxmox uses a self-signed certificate, so the client pins its fingerprint. It checks the fingerprint right after the secure handshake, and only then adds the token. It also opens a fresh connection for every call. A reused connection skips the handshake, so the check would never run again and the call would hang. A test with the wrong pin shows the server gets zero requests.
The package and the guard
The package is too big for one API call, so the updater sends it in base64 chunks and checks the size and hash at the end. It also checks the VM's hostname first. And the deploy lint now flags SSH and the old root address in that script.
Still bugging me
The token is still root inside that one VM. A certificate renewal will break deploys until I update the pin. And onboarding still needs the root key somewhere.