Evals vs tests: what QA actually owns on an AI feature

A product manager asked her QA lead a simple question before a release: "Did we test the new summary feature?" The answer was yes. Every endpoint returned 200, the JSON parsed, the UI rendered. Two days after launch, support flagged 14 tickets where the summary named the wrong customer. Nothing in the suite was broken. Nothing in the suite was measuring quality either.

Short version

• A test checks that your code behaves as written. It passes or fails.
• An eval scores how good model output is across a dataset. It produces a number, not a verdict.
• You need both. Tests protect the plumbing, evals protect the answer.
• Engineering usually owns the eval harness. QA usually owns the dataset, the rubric, and the pass bar.
• Run tests on every commit. Run evals on prompt, model, or data changes, plus a nightly job.
• If nobody owns the pass bar, the eval becomes a dashboard nobody reads.

---

The plain definitions

A test checks a code path. You give it a known input, and you assert a known result. POST /api/summaries with a valid body returns 200 and a JSON object with a summary field. The retry logic fires three times on a 503. The token limit rejects an input over 8,000 tokens with a clear error. Tests are binary. Green or red.

An eval scores output quality across a dataset. A dataset here means a fixed set of inputs with the answers or expectations you consider good, often called a golden dataset. You run your feature over all of it, score each output, and get an aggregate number, for example "84 percent of 200 cases scored acceptable". Evals are continuous. They move up and down.

The mental shortcut: a test asks did the machine do what I built, an eval asks was the answer any good.

Side by side

What tests still cover on an AI feature

People sometimes claim traditional testing does not apply to AI. Most of an AI feature is ordinary software, and ordinary software breaks in ordinary ways. Your test suite should still cover:

• Request and response handling. Status codes, headers, timeouts, retries.
• Schema validation. The model returns JSON, your parser reads it. Assert the shape on every call.
• Error paths. What happens on a 429 rate limit, a 500 from the provider, or a network drop mid-stream.
• Token and size limits. An 80,000-word document should be rejected or chunked, not sent whole.
• Cost and latency guards. A hard cap that stops one request from spending $4.
• UI behaviour. Loading states, streaming text, the copy button, the retry button, the empty state.
• Permissions. A user at [email protected] must not see a summary of another workspace's data.

None of these need a model to be smart. All of them break in production. This layer stays exactly where it was, in your normal pipeline, running on every commit.

What evals cover

Evals cover the part where the model can be technically correct and practically useless.

• Correctness of content. Does the summary name the right customer and the right amount?
• Groundedness. Is every claim supported by the source document you provided?
• Completeness. Did it include the refund date, which the user asked for?
• Tone and format. Is it the length and register you promised?
• Safety and refusal. Does it refuse the requests it should refuse, and not refuse the ones it should answer?

You score these with a mix of methods: exact rules where a rule works, string and number matching for facts, similarity to a reference answer, and a model scoring another model's output against a rubric. The result is a score per case and an average per run.

Who owns what

This is where most teams stall. Split it like this.

Engineering owns the harness. The code that loads the dataset, calls the feature, runs the scorers, stores results, and publishes a report. It is production-grade code and belongs in the repo next to the feature.

QA owns the dataset. Which cases go in, what the expected answer is, which edge cases are represented, and how the set grows. This is judgement work about real user behaviour, and it is the highest-value thing QA does on an AI feature.

QA owns the rubric. The written definition of what "acceptable" means for this feature, in enough detail that two people scoring the same output agree.

QA and product own the pass bar together. "We ship when at least 90 percent of cases score acceptable and zero cases score harmful." Product brings the risk appetite, QA brings the data.

Engineering owns the fix. When an eval drops, the prompt, retrieval, or model config changes. QA confirms the drop is real and reproducible first.

A short version to paste into your team doc:

Tests are owned like any other tests. The eval dataset is a QA artefact, versioned in git, reviewed like code. The eval harness is engineering code. The pass bar is a product decision informed by QA data.

Where each one runs

1. On every commit: unit and integration tests, schema checks, lint. Under five minutes. Blocks the merge.
2. On any prompt, model, or retrieval change: a fast eval over a 50-case subset. Around ten minutes. Blocks the merge if the score drops more than an agreed amount, for example 3 points.
3. Nightly: the full eval over the whole dataset, against the pinned model version. Publishes a trend chart.
4. Before a release: the full eval plus a manual review pass on 30 sampled outputs.
5. In production, weekly: sample real traffic, score it, compare to the offline number. If offline says 92 and production says 71, your dataset does not look like reality.

Step 5 is the one teams skip and later regret.

Common mistakes

Bad: "Our eval passes, so the feature works." Good: "Our eval scores 91 on the dataset, our tests are green, and our production sample scores 88. Ship it."

Bad: Treating an eval score of 87 as a failure because it is not 100. Good: Comparing 87 to last week's 89 and asking what changed.

Bad: Storing the dataset in a spreadsheet one person owns. Good: Storing it as JSONL in git, reviewed in pull requests like any other change.

Two more worth naming. First, no baseline. A score with nothing to compare it to is a number, not a signal. Record the score for every run and keep the history. Second, mixing the two in one job. When a slow, costly eval sits inside the commit pipeline, developers start skipping the pipeline. Keep them separate.

When a bug comes out of an eval

An eval tells you the score dropped. It does not tell you why, and it rarely produces a ticket an engineer can act on. Turn a low-scoring case into a normal bug report with the input, the exact output, the model version, the prompt version, and the retrieved documents attached. If the failure is visible in the product rather than in the harness, a browser-based reporting tool such as Crosscheck attaches the console logs, network requests, and environment details from the page automatically, so the engineer sees the real request instead of a retyped summary of it.

Then add that case to the dataset. Every eval failure that becomes a permanent case is a bug that cannot return quietly.

Frequently asked questions

Do evals replace my existing test suite? No. Evals score output quality, tests protect the code around the model. A feature with great evals and no tests still breaks on a 429 rate limit.

Should a failing eval block a deploy? Block on a drop, not on an absolute number. A run that falls more than a few points below the last accepted score should stop the merge until someone explains it.

Who writes the eval dataset if we have no QA team? Whoever talks to users most, usually support or product. The skill needed is knowing what real inputs look like, not writing code.

How big should the first eval dataset be? Fifty cases is enough to start and small enough to build in a day. Grow it with real failures rather than inventing cases in bulk.

How often should we rerun the full eval? Nightly against a pinned model version, plus on every prompt or model change. Pinning matters, or you cannot tell your change from the provider's change.