Measuring Whether an Improvement Actually Stuck

Why this matters

Most improvements are declared done the moment the change is announced, and that is exactly where they quietly fail. You add the checklist step, change the dispatch rule, retrain the crew, and move on to the next fire. Nobody circles back. Six months later the same problem is back and you cannot say whether the fix was wrong or whether it was simply never followed. An improvement is not real when you decide it, it is real when it is in place and the problem it targeted has actually changed. This is how to check, and how to tell a fix that needs more time from one that needs to go.

Two different failures, two different checks

When an improvement does not deliver, it failed in one of two distinct ways, and they need opposite responses. You cannot fix it until you know which one you have.

• Adoption failure: the change never really happened. The new step is on the checklist but people skip it. The rule exists but nobody enforces it. The training happened but the old habit won. The fix was fine; it just is not being done.
• Effect failure: the change happened but did not work. People genuinely adopted it, faithfully, and the target problem did not budge. The fix was aimed at the wrong cause.

Chase these the wrong way and you make it worse: retraining people on a fix that does not work, or redesigning a fix that would have worked if anyone did it. Always check adoption first, because effect is meaningless until you know the change is actually happening.

First check: did the change get adopted

Before you ask whether it worked, confirm it is being done. This is the step everyone skips.

• Watch the work, do not survey it. "Are you doing the new step?" gets a yes from everyone. Look at the actual jobs, the actual records, the actual trucks.
• Check the leading indicator. A leading indicator is a signal you can see now that predicts the outcome later. If the fix is a new verification step, the leading indicator is whether the step is showing up as done on current jobs, not next quarter's callback rate.
• Expect a dip and a drift. New habits start strong, then drift back toward the old way once attention moves on. Check adoption again after the spotlight is gone, because week-one compliance tells you nothing about week-ten.

If adoption is low, your problem is follow-through, not the fix. Fix the reason it is being skipped: too slow, unclear, no reason attached, or no consequence for skipping it.

Second check: did the target problem move

Once you have confirmed the change is genuinely happening, look at the outcome it was supposed to change. This is the lagging indicator, the result that shows up later.

• Measure the specific thing you aimed at, not general vibes. If the fix targeted a particular callback cause, track that cause, not the whole callback number, which is full of noise from everything else.
• Compare against before. You need some sense of the baseline. Even a rough "we ran a few of these a month" beats no reference point, because without a before you cannot claim an after.
• Give it a fair window. A fix aimed at a problem that surfaces once a month needs several months to show a trend. Declaring victory or defeat after two weeks on a slow-moving metric is guessing.

Needs more time, or needs to go

Here is the call the whole article builds to. A new process that is not obviously working yet is either underbaked or wrong, and telling them apart saves you from two opposite mistakes: killing a good fix too early, and clinging to a dead one too long.

Lean toward more time when:

• Adoption is climbing and the leading indicators are trending the right way, even if the final outcome has not moved yet.
• The metric it targets is slow-moving and the window has been short.
• The crew's feedback is "this is helping" from the people actually doing it.

Lean toward killing or reworking it when:

• Adoption is solid, the window has been fair, and the target problem has not moved at all. That is an effect failure - the fix was aimed at the wrong cause, so go back to root cause, do not push harder on a wrong answer.
• The fix created a new problem as bad as the one it solved. A cure that costs more than the disease is not an improvement.
• Nobody can state what it is for anymore. A change whose purpose has been forgotten is dead weight; retire it.

Reverting a fix that did not work is not failure, it is the system working. The failure is refusing to revert because you are attached to having been right.

Build the recheck in from the start

The reason follow-through dies is that no one owns the check. Prevent it when you make the change, not after you forget it.

• Name the check when you name the fix. "We will look at whether the new step is being done in two weeks, and whether the callbacks dropped in three months." Owner and date, same as any other task.
• Keep it lightweight. A five-minute look at the right leading indicator beats a quarterly report nobody runs.
• Close the loop out loud. Tell the crew what the check showed. A fix confirmed to have worked is what teaches people that improvement is real and worth their effort, not just the flavor of the month.

The mental model to keep: announcing a change and confirming a change are two different acts, and only the second one counts. The shops that actually get better are the ones that go back and look.

References

• Trade-standard practice for continuous improvement and process verification
• U.S. Small Business Administration (SBA), using metrics to manage a small business
• See related: After-Action Review: Learning From a Bad Job; The Difference Between a One-Off and a Pattern
• See related: Knowing When to Kill a Service That Isn't Working