How to test for AI hallucinations, measurably

A billing assistant told a customer their refund policy allowed 60 days. The real policy says 14. The answer was polite, well formatted, and completely invented. The support team found it because the customer quoted it back in a complaint. Nothing in the test suite could have caught it, because the suite only checked that the endpoint returned 200 and the JSON parsed.

Short version

• A hallucination is a claim the model states as fact but cannot support from its sources.
• "Fewer hallucinations" is not a target. Groundedness rate, citation accuracy, and refusal rate are.
• Break each answer into claims, then check each claim against the source text.
• Verify citations really exist and really say what the answer says they say.
• Build an unanswerable test set. The correct answer there is "I do not know".
• Track the numbers per release. A hallucination rate with no history is a number, not a signal.

---

What counts as a hallucination

A hallucination is a statement the model presents as fact that is not supported by the sources it was given, or that is simply false. Fluent, confident, wrong.

Three kinds show up in real products, and they need different tests.

Made-up facts. The model states a policy, price, date, or name that does not exist anywhere. "Your refund arrives within 60 days."

Wrong grounding. The fact exists in the source, but the model attaches it to the wrong thing. The source says the 60-day window applies to hardware; the model applies it to software.

Fake references. The model cites a document, section, ticket, or URL that does not exist, or that exists but says something else. This is the most damaging kind, because a citation makes the reader stop checking.

Three numbers to track

Replace the word "hallucination" with these three metrics. Each one is countable, comparable across releases, and specific enough to act on.

Add one guard metric so you do not optimise into uselessness: over-refusal rate, the share of answerable questions the model wrongly refuses. A model that refuses everything scores perfectly on hallucinations and is worthless.

Groundedness checks, step by step

Groundedness means every factual claim in the answer traces back to the text you provided. This applies to any feature that answers from documents, including retrieval-augmented generation, or RAG, where the system fetches documents and then answers using them.

1. Capture the sources. Store the exact chunks of text sent to the model with every response. If you do not log the sources, you cannot test groundedness afterwards.
2. Split the answer into claims. One sentence often holds two. "Your refund of $42.00 will arrive within 14 days" is two claims: the amount, and the window.
3. Check each claim against the sources. Numbers, names, and dates can be checked with plain string and value matching. Anything else needs a judge model, meaning a second model that reads the claim and the source and decides whether the source supports it.
4. Score the answer. Grounded when all claims are supported. Partly grounded when at least one is not. Not grounded when the main claim is unsupported.
5. Record the unsupported claims themselves. The list of what got invented is more useful than the score. Group them and you will find patterns, for example that dates get invented far more often than amounts.

A practical shortcut for step 3: run the cheap checks first. Extract every number, date, and proper noun from the answer with a regular expression and confirm each appears in the sources. This catches a large share of hallucinations for almost no cost, before a judge model sees anything.

Bad check: "Does the answer look consistent with the documents?"

Good check: "Claim: the refund window is 14 days. Quote the exact line in the sources that states this, or reply NOT SUPPORTED."

Verifying citations

If your feature shows citations, test them. Users trust cited answers more, which means a wrong citation does more damage than a wrong plain answer.

Run three checks on every citation.

1. Does it exist? Resolve the document ID or URL. A 404 on https://docs.example.com/policies/refunds-v3 is an automatic fail. This check is pure code and should run on every response.
2. Is it the right document? The cited document must be one of the sources actually retrieved for that answer. Models sometimes cite plausible-sounding documents from memory.
3. Does it support the sentence? Take the sentence the citation is attached to and the cited passage, and check support. Numbers and names can be matched directly; the rest needs a judge.

Log a per-answer citation score, for example 3 of 4 citations verified. Then alert when the rate drops. A retrieval change that starts returning the wrong chunks shows up here first, usually before anyone notices the answers got worse.

Building a hallucination test set

You need a set of cases designed to make the model invent things. Your normal golden dataset will not do this, because it is built from questions your product answers well.

Include five groups.

1. Unanswerable questions. The answer is genuinely not in the sources. "What is the refund policy for enterprise plans?" when no enterprise policy document exists. Expected behaviour: the model says it does not know or asks for more information. This group should be 20 to 30 percent of the set.

2. False premise questions. The question assumes something untrue. "Why does the free plan include priority support?" when it does not. Expected behaviour: correct the premise, do not answer around it.

3. Near-miss questions. The sources contain something similar but not the same. The docs cover refunds for hardware; the question asks about software. Expected behaviour: state the distinction rather than transfer the rule.

4. Detail-heavy questions. Questions whose answers need exact numbers, dates, or names. These are where invented specifics appear. Keep the expected values in the case so you can check them exactly.

5. Long-context questions. The answer sits in a long document, far from the start. Models more often drift when the relevant text is buried.

Fifty to eighty cases across these groups is enough to start. Tag each case by group. When the groundedness rate falls, the group breakdown tells you where.

Sourcing tip: the best cases come from real failures. When a tester or a support agent catches an invented answer, capture the whole interaction, not a paraphrase. The exact question, the exact answer, the sources retrieved, and the model version. A browser-based reporting tool such as Crosscheck records the console logs, network requests, and environment details from the page as the report is filed, so the underlying request and response arrive with the ticket and can be turned into a test case the same day.

Reducing hallucinations once you can measure them

Measurement comes first because every fix below needs a before-and-after number.

• Improve retrieval before touching the prompt. Most grounding failures in RAG systems are retrieval failures. If the right chunk never reached the model, no prompt will save it. Check retrieval recall separately: for each test question, was the chunk containing the answer in the retrieved set?
• Give the model an exit. Add an explicit instruction that "not found in the provided documents" is an acceptable and expected answer. Many models invent because the prompt implies they must answer.
• Ask for quotes. Requiring the model to quote the supporting line before writing its answer raises groundedness noticeably and makes automated checking easier.
• Cap the scope. A model asked for a two-sentence answer invents less than one asked for a thorough explanation.
• Add a verification pass for high-risk answers: a second call that checks the draft against the sources and strips unsupported claims. It doubles the cost, so reserve it for answers about money, health, or legal terms.

After each change, rerun the full set and compare all four numbers, including over-refusal. Groundedness up 6 points with over-refusal up 15 points is not a win.

Frequently asked questions

Can I detect hallucinations without the source documents? Only weakly. Without sources you can check internal consistency and known facts, but you cannot verify grounding. Log the retrieved sources with every response so the check is possible.

Is a zero hallucination rate realistic? No. Aim for a low, stable, measured rate on your own test set, with hard limits on the answers that carry real risk, such as prices, policies, and dates.

How often should I run these checks? Groundedness and citation checks on every release and nightly. Cheap number and name matching can run on every production response, since it costs almost nothing.

Does a lower temperature stop hallucinations? It reduces variety, not invention. A model confidently repeating the same made-up policy every time is still hallucinating.

Who should own the hallucination test set? QA, working with the people who know the domain. The hardest part is knowing which questions have no answer in the sources, and that is domain knowledge, not engineering.