Change failure rate shipped with no deploy log behind it

bar chart

Three fixes after one feature (constructed example)

Illustrative, not measured. The first two fixes fall inside the window and count; the third does not.

First fix1hours after feature

Counted: the feature is inside the window.

Second fix2.5hours after feature

Counted: the feature is still inside the window.

Third fix5.5hours after feature

Not counted: no non-fix change remains in range.

Showing a change failure rate, the delivery dashboard has no deploy log behind it. Nothing records what shipped, and nothing ties a problem to the change that caused it. All it has is merged pull requests: the title, the branch, the author and the time.

So every merged pull request counts as a deployment, and a failure is one whose title looks like a repair. I went back through the diff to see what "looks like" means.

Three words and a clock

Each merged pull request is checked against three rules, in order, and the first match wins:

• "revert" in the title counts, on any branch.
• "hotfix" in the title counts if it went into the main branch.
• "fix" in the title counts if it went into main and the same person merged something that wasn't a fix a few hours earlier.

The third rule is meant to be careful. Normal bug fixes aren't failures. A fix right after your own feature probably is. Good idea, loose code.

"Fix" is a substring

The code checks whether the letters appear anywhere in the title. "Add prefix support" contains them. So does "Move test fixtures". Either one can count as a failure. A real repair called "Repair the retry loop" never counts, because it skips the word.

The same check runs on the earlier pull request, so a feature with "prefix" in its name gets treated as a fix and can't trigger the rule. The errors pull both ways. If you build this, match whole words:

const saysWord = (t, w) => new RegExp(`\\b${w}(es|ed|ing)?\\b`, "i").test(t);

Any earlier feature, not the last one

The rule asks whether any feature by that person sits in the window, not just the most recent merge. One feature followed by three quick fixes counts as up to three failures, as long as the feature is still in the window. That's on purpose, and I'm not sure I agree with it.

The two halves don't match

The hotfix and fix rules only look at the main branch. The total they're divided by counts merges to every branch. Side-branch work makes the rate look lower than it should.

The count also lands on the repair, not on the change that broke things. Put three repairs in one pull request and the rate drops. An empty period shows zero, which looks just like a clean one. The dashboard also throws away which rule flagged each failure.

Time to restore has the same habit

The sibling metric calls an issue an incident if its title has words like "down". "Dropdown" has that word in it too.

Reading it anyway

A revert is a fairly honest signal, and one team's trend over a few months still means something. Comparing two teams who name their work differently doesn't.