Back to Feed
article 6m read

Two Keys on the Ring: Moving a Build Signer Without a Flag Day

How a release pipeline replaced a single pinned signing key with a pinned key ring, so a build signer could move to new hardware with an overlap window, recorded provenance and a CI ratchet against trust drift.

Article DepthPractice

Flow

Rotation rollout, step by step

  1. 1Freeze meaning

    Do not change what a valid receipt binds in the same change

  2. 2Add the ring

    Manifest lists the current and replacement keys

  3. 3Ratchet in CI

    Every copy of the pins and every package must match the manifest

  4. 4Shadow

    Replacement builds the same commits; compare receipts and track a match streak

  5. 5Promote

    The replacement may now sign real releases

  6. 6Remove old key

    One reviewed change; receipts it signed stop counting

You have a pipeline where a builder signs a receipt and verifiers at each hop refuse to release anything without it. Now you need to move the signer from the current builder to a replacement builder. Here is a practical path for doing that with an overlap window, using a pinned key ring instead of a single pinned key.

Decide what stays fixed

Before touching keys, write down what the receipt must bind and promise not to change it in the same commit. A rotation that also loosens a binding is two changes hiding as one. If the meaning of a valid receipt is frozen, reviewers only have to reason about who can sign, not what a signature means.

Step 1: introduce a manifest and parse it strictly

Keep the public keys as files and list them in a manifest. Parse it strictly: a trust root should not tolerate surprises, and a file name from data should never be able to point outside the ring's own directory.

import { createHash, createPublicKey } from 'node:crypto'
import { readFileSync } from 'node:fs'
import { resolve, sep } from 'node:path'

type RingEntry = { name: string; file: string; fingerprint: string }

const NAME = /^[a-z][a-z0-9-]{0,40}$/
const FILE = /^[a-z0-9][a-z0-9-]{0,60}\.pem$/ // no separators, no dot segments
const HEX = /^[0-9a-f]{64}$/

function parseEntry(raw: unknown): RingEntry {
  if (!raw || typeof raw !== 'object' || Array.isArray(raw)) throw new Error('ring entry must be an object')
  const fields = Object.keys(raw).sort().join(',')
  if (fields !== 'file,fingerprint,name') throw new Error('ring entry has missing or extra fields')
  const { name, file, fingerprint } = raw as Record<string, unknown>
  if (typeof name !== 'string' || !NAME.test(name)) throw new Error('ring entry name is invalid')
  if (typeof file !== 'string' || !FILE.test(file)) throw new Error('ring entry file name is invalid')
  if (typeof fingerprint !== 'string' || !HEX.test(fingerprint)) throw new Error('ring entry fingerprint is invalid')
  return { name, file, fingerprint }
}

export function loadKeyRing(ringDir: string, manifestText: string, maxKeys: number, keyType: string) {
  const root = resolve(ringDir)
  const manifest: unknown = JSON.parse(manifestText)
  const keys = (manifest as { keys?: unknown })?.keys
  if (!Array.isArray(keys) || keys.length < 1 || keys.length > maxKeys) {
    throw new Error('ring manifest must list between one and maxKeys keys')
  }
  const entries = keys.map(parseEntry)
  for (const field of ['name', 'file', 'fingerprint'] as const) {
    const values = entries.map((entry) => entry[field])
    if (new Set(values).size !== values.length) throw new Error(`ring manifest repeats a ${field}`)
  }
  return entries.map((entry) => {
    const path = resolve(root, entry.file)
    if (!path.startsWith(root + sep)) throw new Error(`ring key ${entry.name} resolves outside the ring directory`)
    const text = readFileSync(path, 'utf8')
    if (createHash('sha256').update(text).digest('hex') !== entry.fingerprint) {
      throw new Error(`ring key ${entry.name} does not match its fingerprint`)
    }
    if (createPublicKey(text).asymmetricKeyType !== keyType) throw new Error(`ring key ${entry.name} has the wrong type`)
    return Object.freeze({ ...entry, publicKey: text })
  })
}

Choices worth copying: an exact field set per entry, a restrictive file-name pattern plus a resolved-path check (belt and braces against traversal), a cap on ring size so it cannot quietly grow, and fingerprints over exact bytes rather than a normalized form.

Step 2: identify the signer, then verify as before

If your receipts already carry a key id, use it, but still confirm the id is in the ring. If they do not, trial-verify. Demand exactly one match, then hand that single key to your existing verification function. The old function stays the authority for bindings; the new code only chooses which key to give it.

Step 3: record the signer downstream

Add the signer's name and fingerprint to whatever the verifier signs or stores next. When a later hop re-verifies, it identifies the signer on its own and rejects the artifact if the recorded values differ. "Which builder produced this release" becomes a fact you can query instead of a guess.

Step 4: add a drift ratchet to CI

The quiet failure in multi-component trust is one copy that missed the update. Write a check that gathers every copy of the trust list and compares it to the manifest. Here is the shape of the recommended tests:

import { test } from 'node:test'
import assert from 'node:assert/strict'
import { findTrustDrift } from './trust-drift.js' // your own ratchet

test('a consumer with a stale pin list fails the ratchet', () => {
  const manifest = [{ name: 'builder-a', fingerprint: 'aa'.repeat(32) }]
  const consumers = { 'verifier-one': [{ name: 'builder-a', fingerprint: 'bb'.repeat(32) }] }
  assert.deepEqual(findTrustDrift(manifest, consumers), ['verifier-one: pins differ from the manifest'])
})

Have it cover three things: embedded pin copies, fingerprint checks in whatever installs keys, and the file lists of whatever packages trust material. A package that stops shipping one key is as dangerous as a wrong pin.

Step 5: shadow the replacement, then promote

Let the replacement builder build the same commits as the current builder without any authority. Compare the two receipts field by field and track a streak of consecutive matches, plus build time, as the promotion bar. Generate the replacement's private key on that builder and only publish its public half for review.

Step 6: remove the old key as its own change

Delete the old entry from the manifest and every copy, then reinstall wherever trust material lives. In a ring design, removing a key invalidates every receipt it signed, so drain anything in flight that depends on one before you merge.

Tradeoffs

  • Trial verification costs one signature check per key. With a small capped ring that is negligible, but it is a reason to cap the ring.
  • Two signers means two things to protect. Keep the overlap short and write its exit criterion down before it starts.
  • Embedded pin copies duplicate the manifest on purpose. The duplication makes a trust change visible in each component's diff, and the ratchet keeps the duplicates honest.

How to verify it worked

Recommended tests: feed each verifier a receipt from the current key, one from the replacement key, one from an unlisted key, and a manifest that lists one key twice or names a file outside the ring directory. You want accept, accept, reject, reject, reject, with the signer recorded on both accepts.

Keep reading

Pick up a connected idea or branch into a nearby one.

5 paths forward