Back to Feed
article 3m read

Plans could be rejected for coming from the future

Reviewed purchasing plans were stamped by the database clock and checked against the app server's clock, so a few seconds of skew could make a brand-new plan look like it hadn't been made yet. The authority checks now read the database clock once, in the same transaction (right under the locks, for approval), with no fallback.

Article DepthPractice

Flow

Making stamp and check use one clock

  1. 1Find the comparisons

    List each place a stored time meets the current time, and which clock made each.

  2. 2Pick the stamping clock

    Ask the clock that stamped the data, here the database.

  3. 3Read once, inside the transaction

    Take the wall-clock time inside the transaction, under the lock where there is one, and pass that one value down.

  4. 4Stop on a bad read

    Raise an error instead of using the app server clock.

  5. 5Print both times

    Show the stamp, the check time and the age in every refusal.

  6. 6Test with a slow clock

    Fake only the date, a few seconds behind, and expect fresh plans to pass.

Thrown out seconds after being made: that could happen to a plan in the ordering system, because by one of its clocks the plan hadn't been made yet.

The stamp and the check

A buyer reviews a plan before it becomes a purchase order. When the plan is made, it gets stamped with a time. When someone approves it, a check makes sure the plan is only a few minutes old and still from today. Old plans shouldn't turn into orders. That part is good.

Two clocks

The stamp came from the database. The check asked the app server what time it was. Those two clocks are never exactly equal. If the app server runs a few seconds behind, a fresh plan seems to come from the future, and the check refuses it. A rule meant to catch old plans was rejecting new ones.

The same habit showed up elsewhere. The material review read the app clock when the buyer started a review and again when they accepted it. Receipt reconciliation looked up records "as of now" using the app clock, while those records were stamped by the database. With the app clock behind, my read is that it could miss records that were really there and raise a blocker claiming the data couldn't be vouched for. Two very different errors, one cause.

How each step reads the time now

Each of those steps now asks the database for the time, because the database stamped the data. It reads that time inside the same transaction (in approval, right after taking the locks), and uses the wall-clock time at that moment, not the time the transaction started. Each step reads it once and passes that one value to everything that needs it. If the read fails, the step doesn't go ahead, and it never falls back to the app server's clock, since that would just bring the bug back. And the refusal now prints both times and the age: a negative age means the clocks disagree, a big positive age means the plan really is old. If you stamp something in one place and check its age somewhere else, that's the part I'd copy.

Making the skew happen on purpose

I don't have a real refusal from production to show. I have tests that cause it on purpose. Fake only the date, not the timers. I kept the timers real so the database driver keeps working. Set the fake date a few seconds behind the plan's stamp, run approval and review, and expect them to pass. Keep the real clock for tests that are supposed to fail, so they fail for the right reason. If your unit tests mock the raw database query, make the mock return a time too.

Watch your test data as well. My fixtures were stamping plans with the app clock, and I'd guess that's how this showed up. Some test records still get the app clock, so I wrote a small helper that waits out a small gap before the next step. It checks the record first and never changes it.

When the records come from more than one place

All of this only works when a single database stamps every record involved. If your records come from more than one database, or from outside systems with their own clocks, asking the database won't settle it, and you'll need a different answer.

Keep reading

Pick up a connected idea or branch into a nearby one.

5 paths forward
Continue the thread

The draft button checked for duplicates, then left a gap wide enough for a second one

The button that saves a delivery as a draft in the stores' inventory system checked first and wrote its record only after a slow outside call, so a second click could slip through. Now it claims the delivery in the database first, lets a partial unique index refuse the second claim, fixes a payload hash that couldn't see nested fields, and adds a narrow admin cancel for claims that get stuck.

via #databases · #reliability