Pairwise testing: 400 cases down to 20 without losing coverage

A team wanted to test their dashboard across five browsers, five operating systems, four user roles, and four pricing plans. That is 400 combinations. At fifteen minutes each, more than twelve working days for one release.

So they picked ten combinations that felt sensible and shipped. The bug that reached customers was Safari on iOS with a Viewer role, which nobody had tried together.

Pairwise testing is a way of choosing those combinations on purpose instead of by feel.

Short version

• Most configuration bugs are caused by two settings clashing, not five.
• Pairwise testing covers every pair of values at least once, and ignores larger combinations.
• A free tool called PICT generates the list for you in seconds.
• A 400-combination matrix comes down to about 20 test cases.
• Add rules so the tool never asks you to test Safari on Windows.

---

The idea in plain English

Look at any bug you have seen in a configuration matrix. Almost all of them sound like this: "It breaks in Safari when the user is a Viewer." Two things. Not four.

Studies of real defect data have found the same pattern repeatedly. A large share of failures are triggered by a single value, and most of the rest are triggered by a pair of values. Failures that need three or more specific values together are much rarer.

Pairwise testing (also called all-pairs testing) takes advantage of this. Instead of testing every combination, you build a smaller set of test cases with one guarantee: every possible pair of values from any two settings appears together in at least one test.

You give up coverage of three-way and four-way combinations. In exchange, the test count drops from hundreds to tens.

Why the numbers work out so well

Here is the part that surprises people. One test case covers many pairs at once.

Take the test "Chrome + macOS + Admin + Team plan". That single row covers six pairs:

• Chrome + macOS
• Chrome + Admin
• Chrome + Team
• macOS + Admin
• macOS + Team
• Admin + Team

With four settings, every test case knocks out six pairs. The generator's job is to arrange the values so that as few tests as possible cover all the pairs, with as little overlap as it can manage.

The number of test cases you need grows with the size of the two largest settings, not with the total number of combinations. That is why adding a fifth setting barely changes the answer.

The model

Write down the settings and their values. In pairwise language, a setting is a parameter and each option is a value.

• Browser: Chrome, Firefox, Safari, Edge, Samsung Internet
• Operating system: Windows 11, Windows 10, macOS 15, iOS 18, Android 15
• Role: Viewer, Editor, Admin, Owner
• Plan: Free, Pro, Team, Enterprise

That is 5 x 5 x 4 x 4 = 400 combinations.

Two habits make models better. First, keep the values to the ones that actually change behaviour — five browsers, not the eleven you could name. Second, if a setting has ten values and only three matter, use equivalence partitioning to group them first, then feed the groups into the pairwise model.

Adding rules so the output makes sense

A plain generator will happily tell you to test Safari on Windows 11. That combination does not exist, so you need constraints: rules that tell the tool which combinations to skip.

PICT is a free, open-source command line tool from Microsoft that does this. Put the model in a plain text file, matrix.txt:

Browser: Chrome, Firefox, Safari, Edge, Samsung Internet
OS:      Windows 11, Windows 10, macOS 15, iOS 18, Android 15
Role:    Viewer, Editor, Admin, Owner
Plan:    Free, Pro, Team, Enterprise

IF [Browser] = "Safari" THEN [OS] IN {"macOS 15", "iOS 18"};
IF [Browser] = "Samsung Internet" THEN [OS] = "Android 15";
IF [Browser] = "Edge" THEN [OS] IN {"Windows 11", "Windows 10", "macOS 15"};
IF [Browser] = "Firefox" THEN [OS] <> "iOS 18";
IF [OS] = "iOS 18" THEN [Browser] IN {"Chrome", "Safari"};

Read the last rule out loud: if the operating system is iOS 18, the browser must be Chrome or Safari. That is all a constraint is.

Running the generator

1. Download PICT from its GitHub releases page, or install it with a package manager such as Homebrew.
2. Save the model above as matrix.txt.
3. Run pict matrix.txt in a terminal.
4. Redirect the output to a file you can open in a spreadsheet: pict matrix.txt > cases.tsv.
5. Open cases.tsv. It is tab separated, so any spreadsheet will read it.

For three-way coverage instead of pairs, run pict matrix.txt /o:3. Expect roughly three to four times as many rows.

The output

PICT returns about 20 rows for this model. Here are the first ten:

Row 10 is the combination that reached customers in the opening story. No human picked it. The generator produced it because Safari and Editor had not yet appeared together.

PICT uses a random seed, so your run may produce a row or two more or fewer than someone else's. Use the /r:1 option to fix the seed if you want the same list every time, which is helpful when the output is checked into a repository.

What pairwise does not catch

Be honest about the trade-off.

Three-way bugs. If a failure only happens with Safari and iOS and the Enterprise plan, a pairwise set may miss it. Run those specific combinations separately if you have a reason to suspect them.

Order and history. Pairwise says nothing about sequence. Use state transition testing for flows.

Values inside a value. "Chrome" covers many versions. Pairwise treats it as one thing.

Combinations that must always run. Your top three customer configurations should be tested every release regardless of what the generator says. PICT lets you force these in with a seeding file passed via /e:seedfile.txt.

Fitting it into a real test plan

1. Build the model once and keep it in the repository next to the tests.
2. Generate the cases and paste them into your test management tool as a data table.
3. Mark the two or three must-run configurations as seeds so they always appear.
4. Regenerate whenever a browser, plan, or role is added or removed.
5. Record results against the row number, so "row 10 failed" means something to everyone.

When a pairwise run does find a failure, the configuration is the most important part of the report. Write the full row, not just the browser. A browser-based reporting tool such as Crosscheck captures the browser, operating system, screen size, console errors, and network calls from the page itself, which removes the risk of the row being copied down wrong.

Frequently asked questions

Is pairwise testing safe to use on critical features?

Use it for configuration coverage, not as your only technique. Keep full coverage of the specific combinations that carry money, permissions, or safety consequences.

What is the difference between pairwise and orthogonal array testing?

Orthogonal arrays are a stricter mathematical structure where every pair appears the same number of times. Pairwise only requires each pair to appear at least once, so it usually produces fewer tests and handles constraints better.

Which generator should I use?

PICT is free, small, and handles constraints well. ACTS from NIST is also free and supports higher-strength coverage. Many test management tools now include a generator too.

How many parameters can I include?

Dozens. The test count is driven mainly by the two largest parameters, so adding a fifth or sixth setting usually adds only a few rows.

Can I use pairwise for form fields rather than configurations?

Yes. Partition each field first, then feed the partitions in as values. It works well for forms where several fields interact, such as plan, billing period, currency, and tax region.