Testing a RAG system: retrieval, grounding, and citations

A support assistant at an insurance company told a customer that cancelling a policy returns the full premium. The real policy document says 70 percent, minus a fee. The model did not make this up. It read a document called policy-terms-v3.pdf, which was replaced in March but never deleted from the search index.

That is a retrieval bug wearing a hallucination costume. If your test plan only reads the final answer, you will keep filing tickets against the wrong component.

Short version

• RAG means the system searches your documents first, then asks a language model to answer using what it found.
• Test retrieval and generation separately. They break for different reasons and get fixed by different people.
• A retrieval test needs a fixed set of questions with known correct documents. Build that first.
• Grounding tests ask one thing: is every claim in the answer supported by the retrieved text?
• Stale index and wrong-version bugs are the most common real-world failure, and the easiest to test for.
• Citations are a feature. Test that they point at the passage that actually contains the fact.

---

What RAG actually is

RAG stands for retrieval-augmented generation. In plain terms: before the model answers, the system searches a collection of your documents, pulls out the few chunks that look most relevant, and pastes them into the prompt.

A chunk is a small slice of a document, usually a few hundred words. Documents get cut into chunks because a model can only read so much at once.

The pipeline has four steps:

1. Your question is turned into an embedding — a list of numbers that represents its meaning.
2. A vector database finds the chunks whose embeddings are closest to yours.
3. The top chunks (often 3 to 10) are inserted into the prompt.
4. The model writes an answer using those chunks.

Each step can fail on its own. Step 2 can return the wrong chunks. Step 4 can ignore correct chunks and answer from memory instead.

Split the system into two testable halves

Treat retrieval and generation as separate features with separate test suites.

The value of this split shows up in triage. "The bot gave the wrong refund amount" tells nobody where to start. "Retrieval returned policy-terms-v3.pdf, the 2023 version" makes the fix obvious in a minute.

Make your bug template ask for the retrieved chunk IDs. If your app does not expose them, add a debug view behind a flag.

Build a retrieval test set first

You cannot measure retrieval without a list of questions and their correct sources. Start small. Fifty questions beat zero questions, and you can write fifty in an afternoon.

For each row, record:

• The question, in the words a real user would use.
• The document ID or file name that contains the answer.
• The exact sentence that answers it, copied and pasted.

An example row:

Question: How long do I have to cancel a new home policy? Correct source: policy-terms-2026-04.pdf, section 4.2 Answer sentence: "New home policies may be cancelled within 14 days of the start date for a full refund."

Ask support and sales for real questions. Real users write "can i cancel", not "What is the cancellation window for a newly issued residential policy?" Both belong in the set, but the messy one finds more bugs.

Measure retrieval on its own

Once you have the set, run only the search step and compare the returned chunk IDs against your expected ones. Three numbers cover most needs.

MRR stands for mean reciprocal rank. If the right chunk is first, that question scores 1. Second scores 0.5. Third scores 0.33. Averaging across questions gives you one number for "how near the top".

Set a floor and treat it as a gate. A reasonable starting rule: recall@5 must stay at or above 0.90 on the fixed set, and any single question that flips from pass to fail gets looked at by a human before release.

Test grounding, not just correctness

Grounding means every factual claim in the answer can be traced to the retrieved text. An answer can be correct and ungrounded — the model knew the fact from training and got lucky. That is still a bug, because next time it will guess wrong on your data.

A simple manual grounding check, per answer:

1. Print the answer and the chunks that were retrieved.
2. Underline every factual claim in the answer: numbers, dates, names, conditions.
3. For each claim, find the supporting sentence in the chunks.
4. Any claim with no supporting sentence is an ungrounded claim.
5. Record the count. Your target is zero.

The pattern to watch for is the confident bridge. The chunks say "policies may be cancelled within 14 days" and "a 30 pound admin fee applies to changes". The model writes "you can cancel within 14 days and pay a 30 pound cancellation fee". Nothing there says the fee applies to cancellations. The model joined two true sentences into one false one.

Good and bad answers to the same question:

Bad: "You can cancel any time in the first month and get all your money back." No chunk says "any time in the first month". The real window is 14 days.

Good: "You can cancel within 14 days of the start date for a full refund [policy-terms-2026-04.pdf, 4.2]. After that, refunds are prorated." Every number appears in the retrieved text, and the source is named.

Test citations as a real feature

Users click citations. If a citation opens a document that does not contain the claim, trust drops faster than if you had shown no citation at all.

Check four things for every cited answer:

• The link resolves. No 404, no permission error for a normal user like [email protected].
• The document matches. The cited file is the file the fact came from.
• The passage matches. The specific section or page contains the claim, not just the general topic.
• Every claim that needs a source has one. Missing citations are as bad as wrong ones.

A cheap automated version: assert that the cited chunk ID appears in the list of chunks the retriever actually returned. It catches the worst class of bug, where the model invents a plausible file name such as policy-terms-2026.pdf that has never existed.

The two bugs you will actually find

Wrong document

Symptoms: the answer is confidently wrong, and the numbers in it are real numbers from somewhere else. Usually the index contains several documents that look alike — a template, a translated copy, a draft, a regional variant.

Test for it by writing questions where two documents both mention the topic but only one is correct. Ask about the UK cancellation window when the index also holds the Ireland version. This is where recall@5 looks fine but the top result is wrong, so watch MRR here.

Stale index

Symptoms: the answer matches an old policy exactly. Everything works, just against last year's truth.

This is the failure nobody writes tests for, because the pipeline is green. Add a freshness check that runs daily:

1. Pick five documents that changed in the last 90 days.
2. For each, find a fact that changed — a price, a date, a limit.
3. Ask the system a question whose answer depends on the new value.
4. Fail the check if the old value comes back.
5. Also assert the index's newest document timestamp is within your expected sync window, for example 24 hours.

Add one more test that most teams skip: delete a document from the source system and confirm it disappears from answers. Many indexing jobs only add and update. Deleted files sit in the vector database forever.

Filing a RAG bug so it gets fixed

A useful RAG bug report carries the question, the full answer, the retrieved chunk IDs, the model and index versions, and the timestamp. Screenshots of the chat alone force the engineer to guess.

If the assistant lives in a web app, a browser-based reporting tool such as Crosscheck captures the screenshot, console logs, network requests, and environment details in one step, which usually means the retrieval API response is already attached before anyone asks for it.

Frequently asked questions

Do I need a golden test set, or can I test with random questions?

You need a fixed set. Random questions tell you nothing about whether last week's change made things better or worse. Keep the same fifty to two hundred questions and rerun them on every index rebuild.

How often should the retrieval suite run?

Every time the index is rebuilt, the embedding model changes, or chunking settings change. Those three events cause almost all retrieval regressions. A daily scheduled run catches slow drift from new documents.

Can I use a language model to grade grounding automatically?

Yes, and it works reasonably well for a first pass. Ask a separate model to list claims in the answer and mark each as supported or unsupported by the given chunks. Have a human review every failure, because the grader itself makes mistakes.

What is a good recall@5 target?

Above 0.90 for a well-curated document set is realistic. Below 0.80 usually points at chunking that splits answers in half, or a document set that never contained the answer at all.

The answer is right but the citation points at the wrong page. Is that a bug?

Yes, and file it as a normal-priority bug rather than a minor one. Users check citations when they doubt an answer, so a wrong citation does its damage at the exact moment trust is already low.