Building a golden dataset for an AI feature

A team spent three weeks tuning prompts for an invoice reader. Every change felt better in the demo. Then a customer sent an invoice with the total in the top-right corner instead of the bottom, and the feature returned "total": null. Nobody had that layout in their test examples. They had 12 examples, all from the same accounting tool, all collected on the same afternoon.

Short version

• A golden dataset is a fixed set of inputs plus the answers you consider correct.
• Start with 50 to 100 cases. Coverage matters far more than size.
• The people who know real users should choose the cases. QA usually owns this.
• Two labellers per case, with a written rubric, and a way to settle disagreements.
• Weight it toward edge cases, not toward the happy path you already know works.
• Refresh it every sprint by adding real failures. Retire cases that no longer reflect the product.

---

What a golden dataset is

A golden dataset is the fixed collection of inputs you run your AI feature against, together with the answer or the criteria you have agreed is correct for each one. Some people call it a golden set, an eval set, or reference data. The name matters less than the property: it is stable, so a score from today can be compared to a score from last month.

Each entry usually holds five things:

{
  "id": "inv-0042",
  "input": "s3://qa-fixtures/invoices/acme-2026-03.pdf",
  "context": "vendor=Acme Ltd, currency=EUR",
  "expected": { "total": 1284.50, "currency": "EUR", "due_date": "2026-04-15" },
  "tags": ["layout:total-top-right", "currency:eur", "source:production"]
}

The tags field does more work than it looks. It is how you later answer "how do we score on non-English inputs" without building a second dataset.

How many examples do you need

The honest answer is: enough that adding ten more cases stops changing your score much. In practice, use these bands.

Two rules keep the size honest. First, a dataset of 300 near-identical cases is worth less than 60 varied ones. Second, if you cannot run the full set in under an hour, you will stop running it. Split it into a fast subset for pull requests and a full set for nightly runs.

Small sets have a statistics problem worth knowing. On 50 cases, a change from 88 percent to 92 percent is two cases flipping. That is noise, not progress. Below about 100 cases, treat anything under a 5 point move as inconclusive and rerun before you celebrate.

Where the cases come from

Rank your sources by how close they are to real users.

1. Production logs. The best source by a wide margin. Sample real inputs across a full week so you catch Monday morning and Friday evening behaviour. Remove or replace personal data before it enters the repo.
2. Support tickets and bug reports. These are pre-labelled failures. Someone already decided the output was wrong and often wrote why.
3. Sales and onboarding calls. The inputs prospects try first, which are often the weirdest.
4. Written by the team. Fine for filling gaps in coverage. Bad as your main source, because you will write the inputs your feature already handles.
5. Generated by a model. Useful for bulk variations on a case you already have. Never trust it for the expected answer without a human check.

The bug report pipeline deserves a mention here. When a tester finds a bad AI output in the product, the case is only reusable if the report captured the exact input, the exact output, and the environment. A browser-based tool such as Crosscheck records the console logs, network requests, and environment details alongside the report, which means the failing model call arrives with the ticket and can be lifted straight into the dataset.

Which edge cases to include

Aim for roughly 40 percent happy path and 60 percent edges. That ratio looks wrong until you remember that the happy path is the part that already works.

Work through this list for your own feature.

• Empty and near-empty input. A blank document, a one-word question, a form with only a name.
• Very long input. Something near your token limit, and something over it.
• Wrong language. A German invoice in an English-only flow. A mix of two languages in one message.
• Ambiguous requests. "Cancel it" with three open orders. The right answer is often a clarifying question, not a guess.
• Missing information. A question whose answer is genuinely not in the source. The correct behaviour is to say so.
• Conflicting information. Two documents that disagree on the price.
• Formatting oddities. Tables inside tables, scanned images, emoji in names, a customer literally called Null.
• Numbers and units. 1,284.50 versus 1.284,50. Kilograms versus pounds. Time zones.
• Adversarial input. Text that tries to change the model's instructions.
• Cases that must be refused. Requests for medical dosing, legal advice, or another user's data.

Tag each of these. When a release drops 6 points, the first useful question is which tag dropped.

Who labels, and how

Labelling means writing down the correct answer, or the criteria for a correct answer, for each input. This is the part teams underestimate.

Choose labellers who know the domain. For an invoice reader, someone from finance beats a smart engineer. For a support summariser, a support agent beats both.

Write the rubric before you label. One page. Define acceptable, unacceptable, and borderline, with two examples of each. Without this, two people label the same output differently and your score becomes meaningless.

Bad rubric line: "The summary should be good and helpful."

Good rubric line: "Acceptable: names the customer, states the problem in one sentence, and lists any amount mentioned. Unacceptable: invents an amount, names the wrong customer, or exceeds 80 words."

Use two labellers on every case. Then measure how often they agree. If two people agree on fewer than about 80 percent of cases, the problem is the rubric, not the people. Rewrite it and relabel.

Have a tie-breaker. One named person, usually the product owner, settles disagreements. Record the decision as a new rubric line so the same argument does not repeat.

Label the reason, not just the verdict. "unacceptable" teaches nobody anything. "unacceptable: invented a due date not present in the document" becomes a failure category you can count.

Follow these steps for a first labelling round:

1. Pick 30 cases and give the same 30 to two labellers.
2. Label independently, no discussion.
3. Compare. List every disagreement.
4. Rewrite the rubric to settle each disagreement with a rule.
5. Relabel the disagreements. Then label the rest of the set.

Keeping the dataset current

A golden dataset rots. The product changes, the prompt changes, and cases that were once hard become trivial. Put maintenance on a schedule.

Every sprint. Add every real failure found that sprint. Two to five new cases is a healthy rate. Fewer means you are not looking; many more means something bigger is wrong.

Every quarter. Review the whole set. Retire cases that no longer reflect the product, for example a case about a field you removed. Move cases that now pass 100 percent of the time into an archive slice you run monthly instead of nightly. Check that the mix still matches production traffic; if 30 percent of real users now write in Spanish, your set should reflect that.

On every product change. A new document type, a new language, or a new output field means new cases before the change ships, not after.

Version it like code. Keep the set in git as JSONL. Every change goes through a pull request with a reason in the description. Tag releases, for example golden-v7, and record which version produced every eval score. Comparing a score from golden-v5 to one from golden-v7 is comparing two different exams.

One rule protects everything above: never edit the expected answer to make a failing run pass. If the expected answer really was wrong, fix it in a separate pull request, say so plainly, and rerun the baseline. Quietly relaxing an expectation to get a green build is how a dataset stops meaning anything.

Frequently asked questions

Can I use production data directly in my golden dataset? Only after removing personal data. Replace real names, emails, and account numbers with stable fakes such as [email protected], and keep the structure so the case still tests what it tested before.

What if there is no single correct answer? Store criteria instead of an answer. For open-ended output, the expected field can be a list of facts that must appear and a list of things that must not, scored by rules or by a judge model.

How do I stop the dataset from becoming one person's private project? Keep it in the repo, require a review on changes, and report the eval score in the same place the team reads test results. Shared visibility is what makes it shared work.

Should the dataset include cases the feature currently fails? Yes. Known failures belong in the set, marked as known. Removing them hides the gap, and you lose the signal on the day someone fixes it.

How do I know my dataset is good enough? Compare offline scores to a sample of scored production traffic. If they track within a few points, your set looks like reality. If offline is much higher, your set is too easy.