Back to Feed
article 3m read

Any copy holding an API key could email my suppliers

Any copy of my ordering system that could see an API key sent supplier email. I moved the keys into the database and added a release gate that ships with each version, so a dev copy holding a real key still can't act.

Article DepthPractice

Flow

Rollout path for an existing app

  1. 1Inventory effects

    List call sites and group them into coarse capabilities; record unguarded ones in a baseline.

  2. 2Write policy as data

    Ordered rules, default off, missing attribute never matches, one-way kill switch.

  3. 3Move secrets

    Stored credentials, database-only resolvers, typed absence, cache cleared on write.

  4. 4Guard choke points

    Assert before claims and logs; pass store as a request attribute where scoped.

  5. 5Ratchet and audit

    Fail the build on new unguarded sites; print resolved state per channel.

Just one question ran in the ordering system's email code before it emailed a supplier: is there an API key in the environment? If yes, send. If no, stop.

That code couldn't tell production from one of my dev copies. A key shows the app can log in to the email service. It doesn't say this copy of the app should be sending anything. Email, the task tracker, the point-of-sale gateway and the shelf-label system all worked this way. Copies of an app spread through restores and copied settings files, and any copy holding a real key could act with it. I had no incident to point at. I didn't want to wait for one.

So the agent and I split the question in two. Both halves have to say yes.

Keys come from one place now

The email and gateway keys moved into an encrypted table in the database. The code that fetches them never looks at environment variables, so a key left in the environment is ignored.

A missing key now raises its own error that tells you how to set it. Code that can live without a key catches only that error, so real failures still show up as failures. Keys are cached for about a minute, saving a key clears the cache, and each lookup records when the key was last used.

Permission ships with the release

The second half is a release gate: a small policy file that ships with each release and says what this copy of the app may do. If you're building something similar, these are the rules I'd copy:

  • Match on facts about the deployment, like the release channel, the server and the domain, plus request facts like a store.
  • Let the first matching rule win, and default to no.
  • If a rule names something the request doesn't have, the rule doesn't match. That includes "everything except" rules.
  • Treat an unknown feature or a broken policy file as no.
  • Give yourself a kill switch that can only turn things off.

The gate runs before the key lookup, so a blocked send never touches the secret. At launch, email was on only in production, and live writes to the stores were off everywhere.

Where the gate sits

Every email send has to say which feature it belongs to, and the compiler checks it. A blocked send gets a plain "not available" answer, not a crash. Order sending checks the gate before it validates or claims the order, so a blocked send leaves nothing half-done.

Store tag writes are checked one store at a time. In dev, only a short list of stores I picked is open. In production the gate says yes and the existing write switches decide, so the gate can't loosen production.

Keeping it from rotting

A build check looks for code that touches the outside world without the gate. The first run found about twenty spots, and they went on a list that can only get shorter. New ungated code fails the build. The email work cleared most of that list.

The check only matches patterns, so sneaky indirect calls can still hide. And as far as I can tell, a blocked send in production only shows up in the logs. I haven't decided what should wake me up.

Keep reading

Pick up a connected idea or branch into a nearby one.

5 paths forward