Back to Feed
article 3m read

A login was required for CI jobs just to look up each other's names

Private per-job Podman networks on my CI runner ran into a DNS helper that wants a logged-in user's systemd bus. A cgroup detour lasted one change; a tiny stand-in launcher that catches that one request is what stuck.

Article DepthPractice

Flow

The order I did it in

  1. 1Profile rules

    The network helper was allowed to write in each run folder, and later to read one file there.

  2. 2Patient cleanup

    The install check retries deleting its scratch folder, then fails loudly.

  3. 3Cgroup detour

    A locked config per job, removed by the very next change.

  4. 4Wrapper

    A program first on the job PATH runs the DNS helper directly for one exact request.

  5. 5Pipeline check

    Each run of my own pipeline makes a network, runs a container on it, and removes it.

Rootless containers run fine in my CI jobs. But put them on a private network, and the part that lets them find each other by name wants a login the job account never has.

How my jobs run

The runner is my own VM. Each job is a systemd unit that switches to a plain worker account with its own throwaway folders. Containers are rootless Podman, with no Docker socket. Nobody ever logs in as the worker, and I want to keep it that way.

I wanted each job to get its own private network, so a test's database and app can find each other by name and other jobs can't reach them. Podman runs a small DNS helper for that. From what my wrapper catches, Podman starts the helper by asking systemd for a scope in the user's own systemd. That needs a user message bus, which comes with a login. My worker has none.

Profile rules and a patient cleanup

The first changes that day were at the edges. The AppArmor profile for Podman's network helper got a rule to write in each run's network folder, then later a rule to read one file there. In between, the installer's self-test learned to retry deleting its scratch folder for a few seconds, then fail loudly. If you confine these helpers too, expect to add rules one at a time.

A detour I undid

Next I forced Podman onto a different cgroup setting with a locked config file per job. That took a path check, a protected variable and a new self-test. The next change threw it all away except a network test. I don't know what it did when it ran. Probably it just wasn't the setting that decides how the DNS helper starts.

The wrapper

What stuck is a tiny script. The runner puts a root-owned folder first on every job's PATH, holding a program with the same name as systemd's launcher. If it sees the DNS helper's exact scope request, it runs the helper directly. Anything else goes to the real launcher.

Anything that isn't the DNS scope request is handed on to the real launcher unchanged.

I'm okay with that because each job already runs in its own isolated unit. These changes don't show how that unit cleans up when it stops, so check yours before you copy this. The wrapper runs as the job, so it can't give the job any extra power.

The wrapper ships with each release, so it updates with everything else. The install self-test uses the job's PATH. And a pipeline step now makes a network, runs a container on it, and removes it.

The part my pipeline still skips

If Podman changes how it calls the helper, the match fails and the login problem comes back. The unit tests only check that the right lines are in place. The network checks I added run one container that does nothing. So if you copy this, give that pipeline step two containers and a name lookup. Mine is still missing that part.

Keep reading

Pick up a connected idea or branch into a nearby one.

5 paths forward