Defect leakage: measuring what QA missed

A release goes out on Thursday. By Monday, support has logged nine tickets. Three are the same broken filter on the reports page. Someone in the retro asks the obvious question: how did we not catch that?

Defect leakage is the number that answers it. Used well, it points at a gap in your testing. Used badly, it turns into a stick for beating testers with.

Short version

• Defect leakage is the share of bugs that reached users instead of being caught during testing.
• The formula is leaked defects divided by total defects, times 100.
• Count a defect as leaked only if it was findable in your test environment. Data-only and third-party outages do not count.
• Most product teams land between 5 and 15 percent. Under 5 percent is strong. Over 25 percent means a real gap.
• Severity-weighted leakage is more useful than raw leakage. One leaked payment bug matters more than twelve leaked typos.
• Review leakage by cause and by module, never by tester.

---

What defect leakage actually measures

Defect leakage measures containment: how much of the total defect load your testing stage caught before users saw it.

It does not measure how good your testers are. It measures how well the whole delivery process — requirements, development, code review, testing, and release — held bugs inside the building.

You will also see this called defect escape rate or defect containment efficiency. The names differ; the idea is the same.

The formula

Defect leakage = (defects found after release / (defects found before release + defects found after release)) x 100

An example. In the June release:

• Testing found 132 defects before release.
• Users and support found 18 defects in the four weeks after release.

Leakage = 18 / (132 + 18) x 100 = 12 percent.

You will see a second version in older material:

Defect leakage = (defects found after release / defects found before release) x 100

That version gives 18 / 132 = 13.6 percent for the same data. Both are used. Pick one, write the formula in the same place as the number, and never switch. A team that quietly changes the denominator will report an improvement that did not happen.

Two rules that decide whether the number is real

Rule 1: set a fixed measurement window

"After release" needs an end date. If you count production bugs forever, an old release keeps getting worse and you can never compare two releases.

Use a fixed window — 30 days after release is common for web products, 60 or 90 for products with slow upgrade cycles. Bugs found on day 31 belong to the next period's data, not to this release's leakage.

Rule 2: only count what was findable

Not every production bug is a testing miss. Before you count one as leaked, ask whether it could have been found in your test environment with the information available at the time.

That last row matters. Some leakage is a deliberate choice. If you decided not to test Android 9, the bugs that escape there are the price of that decision. Count them, label them "out of test scope", and let the group decide whether the trade is still right.

Realistic benchmarks

Public benchmarks vary wildly because everyone counts differently. Use these as rough bands, not targets:

Two warnings about the low end. A leakage of 1 percent usually means production bugs are being handled in support chat and never entered in the tracker. And a team that ships every two weeks to 50 users will always look better than a team shipping quarterly to 50,000. Compare a team to its own past, not to another team.

Severity-weighted leakage

Raw leakage treats a wrong tooltip and a broken payment flow as equal. They are not.

Report leakage split by severity instead:

The total says 12 percent, which sounds fine. The critical and high rows say one in five serious bugs is reaching users. That is the finding worth acting on, and the single number hid it.

If you track only one figure, track critical and high leakage. Nobody escalates over a leaked typo.

Using the number without blaming testers

The fastest way to make this metric useless is to put it in a performance review. Here is what happens when you do.

Testers start logging more low-severity bugs to grow the denominator. Borderline production issues get argued down to "not a defect". People stop volunteering for the hard, risky modules where leakage is naturally higher. The number improves. Quality does not.

Better practice:

1. Report it per module, not per person. "Reports module: 22 percent leakage" leads to a coverage conversation. "Priya: 22 percent" leads to a defensive one.
2. Attach a cause to every leaked defect. Use a short fixed list so you can count causes.
3. Review the causes monthly. Fix the biggest one. Ignore the rest until next month.
4. Present it with the release context. A rushed hotfix release will leak more. Say so in the same sentence as the number.

A cause list that works for most teams:

• Not covered by any test case
• Covered, but the test case was not run this cycle
• Test case existed and passed, so the case is wrong
• Environment or test data did not match production
• Requirement was unclear or changed late
• Known and accepted before release

The third cause is the most valuable one to find. A test that passes on broken behaviour is worse than no test, because it buys false confidence.

Make leaked bugs easy to trace back

You can only assign a cause if the production report contains enough detail to reproduce it. Support tickets that say "the report page is broken for a customer" cannot be traced to a missing test case.

Push for the same minimum on production reports as on internal ones: the URL, the account, the browser and version, the exact steps, and the console or network errors. Capture tools such as Crosscheck collect the screenshot, console logs, network requests, and environment details in one step from the page itself, which makes an escaped bug analysable instead of anecdotal.

A simple monthly routine

1. Pull every defect created in the last month, split by where it was found.
2. Apply the findable test from Rule 2 and drop what does not qualify.
3. Calculate total leakage and leakage for critical plus high.
4. Group leaked defects by module and by cause.
5. Pick the single largest cause in the largest module and write one change for the next cycle.
6. Show the trend for the last six months next to the current number.

One change per month is enough. Teams that write six actions from one review complete none of them.

Frequently asked questions

What is a good defect leakage percentage?

Most web and SaaS teams sit between 5 and 15 percent. Treat anything under 5 percent with suspicion until you confirm production bugs are being logged properly.

Is defect leakage the same as defect escape rate?

In practice, yes. Both describe bugs that reached users rather than being caught in testing. Confirm which formula your team uses, since the denominator differs between versions.

How long after release should I keep counting?

Pick a fixed window and stay with it. Thirty days works for most web products. Bugs found after the window belong to the next measurement period.

Should defect leakage be part of a tester's performance review?

No. It measures the whole delivery process, and using it on individuals encourages people to game the counts instead of finding bugs.

What if we release every day?

Measure by month or by sprint instead of by release. The window matters more than the boundary, and continuous delivery makes per-release attribution close to meaningless.