A buyer's scorecard for agentic QA tools

Every agentic QA demo goes the same way. The rep types "test the checkout flow" in plain English, a browser opens, and thirty seconds later a green checkmark appears. Everyone in the room nods.

Three months later the same team has 200 tests, a bill nobody predicted, and no idea why test 147 started passing when the feature is clearly broken. The demo was real. The questions were missing.

Short version

• An agentic QA tool is one where an AI agent writes, runs, or repairs tests instead of a person.
• The demo always works. Your job is to find the edges it does not show.
• Twelve questions cover most of the risk: how it heals, what it costs, and whether you can read the tests.
• Score every vendor on the same sheet, in the same order, on the same day if you can.
• The two answers that predict regret most: "we handle that automatically" and "pricing depends on usage".

---

Why a scorecard beats a feature list

Feature lists are written by vendors. Every tool will claim self-healing, AI test generation, and CI integration. The words are the same across all of them, so comparing feature lists tells you nothing.

A scorecard flips it. You bring the questions, you write the answers down, and you compare answers instead of marketing pages. It also slows the call down, which is useful. Vague answers are easier to spot when someone has to give them out loud.

Print the sheet. Ask the questions in order. Score during the call, not after.

The twelve questions

1. When a test "heals" itself, what exactly changed?

Self-healing means the tool repairs a broken test on its own, usually when a button or field moves. Ask them to show you a real healing event from an existing customer, with a before and after.

You want to see something like: the selector changed from #submit-btn to [data-testid="submit"]. If the answer is "the AI figures it out", ask again.

Score high if they can show you a diff. Score low if healing is invisible.

2. Can healing hide a real bug?

This is the question most vendors dislike. If a developer renames the "Pay now" button to "Continue", the test should probably fail — that is a user-facing change. A tool that heals through it has just hidden a bug.

Ask: what is the rule for healing versus failing? A good answer names limits, like "we heal on attribute changes but fail on visible text changes."

3. Can I read the test without your product?

Open the file. Is it code you understand — Playwright, Cypress, plain Python — or a blob of JSON only their runner can read?

Good: a .spec.ts file in our repository, reviewed in the same pull request as the feature.

Bad: a test that exists only in their web app and is described as "step 1: log in".

If you cannot read the test, you cannot review it, and you cannot leave.

4. Can I export everything and run it elsewhere?

Ask for an export of one real suite, in the trial, not a promise. Then try to run it without their runner. Most exports lose something — the assertions, the test data, the setup steps. Find out what.

5. What does one test run cost?

Get a number, not a tier. Ask: for a suite of 50 tests, each about 12 steps, run 20 times a day, what is the monthly bill?

Then ask what happens when a test retries. Many tools retry three times on failure, and each retry may cost the same as a run. A flaky suite can triple your bill without anyone changing a setting.

6. What happens when I go over?

Some tools stop running tests. Some keep running and invoice you. Some silently drop to a cheaper model, which changes your results. All three are survivable; being surprised is not.

7. Which model runs my tests, and can it change without notice?

Ask which model version they use and what happens when the provider updates it. If your test results can change because a vendor swapped models on a Tuesday, you need to know that before you depend on it.

Ask if you can pin a version. Ask how much notice you get.

8. Where does our data go?

Tests touch real screens, and screens contain data. Ask what leaves your network — screenshots, page HTML, console logs, network bodies. Ask whether any of it trains their models. Ask how long it is stored and how to delete it.

Get the answer in writing, not on a call.

9. How does it handle authentication?

This kills more pilots than anything else. Ask them to log in to a staging environment with two-factor authentication enabled, using a real account like [email protected].

Also ask about session reuse. Logging in on every test is slow and often triggers rate limits or account lockouts.

10. What does a failure report look like?

Ask to see a real failure, not a green run. A useful failure report includes the step that failed, a screenshot at that moment, the console error such as Uncaught (in promise) TypeError: Cannot read properties of null (reading 'value'), the failed network calls, and the browser and viewport.

If the report says "test failed at step 7" and nothing else, your team will spend the savings on debugging.

11. How do I stop it from doing something?

Every agent needs a brake. Ask how you exclude a page, block a destructive action like deleting an account, or freeze a test that keeps healing itself wrongly.

12. What does month six look like?

Ask for a customer with a suite older than six months. Ask that customer — not the vendor — how many tests they deleted, how much time they spend maintaining, and whether the bill matched the estimate.

If no such customer exists, the product is young. That may be fine, but price the risk.

The scoring sheet

Score each question 0 to 3. Then multiply by the weight.

Maximum is 90. Use it to rank, not to pass or fail. A tool at 62 that your team will actually use beats a tool at 78 nobody opens.

Two scores act as gates on their own. If question 3 (readable tests) or question 8 (data handling) scores 0, stop there. Everything else depends on those.

Run the same pilot for every vendor

Pick one flow you already know well. A checkout, a signup, a permissions change. Then, for each vendor:

1. Give them the same staging URL and the same test account.
2. Build the same five tests.
3. Break one thing on purpose — rename a button, add a 3-second delay, change a redirect.
4. Record which tools failed, which healed, and which healed when they should have failed.
5. Note the total cost of the pilot from each vendor's dashboard.
6. Ask two engineers who did not build the tests to read one test file and explain what it does.

Step 6 is the quiet winner. If a senior engineer cannot explain the test in 60 seconds, your team will not maintain it.

Agentic tools are good at running flows and weak at judging whether the result looks right to a person. Keep a path for human-reported bugs alongside them — a tester on the page, capturing the screenshot, console logs, and network requests in one action with something like Crosscheck, still catches things no agent flags.

Answers that should slow you down

• "The AI handles that automatically." Ask how, and ask to see it.
• "Pricing depends on your usage." Ask for a worked example with your numbers.
• "You will not need to look at the code." You will.
• "We do not really have flaky tests." Every suite has flaky tests.
• "That is on the roadmap." Buy what exists today.

None of these mean the tool is bad. They mean the answer is not yet real, and you should write "unknown" on the sheet rather than a score.

Frequently asked questions

What does "agentic" actually mean in a QA tool?

It means an AI agent takes actions on its own — writing a test, clicking through a flow, or repairing a broken selector — instead of only suggesting them. The important part is which actions it can take without asking you.

Is self-healing worth paying for?

Sometimes. It saves real maintenance time on cosmetic changes like renamed CSS classes. It becomes a liability when it heals through changes that users would notice, so the value depends entirely on where the vendor draws that line.

How long should a pilot run?

Long enough to include one real release. Two to four weeks is usually the minimum, because most tools look fine until your application changes underneath them.

Should we replace our existing Playwright or Cypress suite?

Not at the start. Run the agentic tool beside your existing suite on the same flows and compare results for a month. If they disagree, that disagreement is the most useful data you will get.

What is the single best question if I only get one?

"Show me a test that healed itself, and the diff of what changed." The answer tells you about transparency, control, and whether the feature is real, all at once.