A postmortem template for an AI feature that shipped a bad answer

A travel app added an AI assistant that answers refund questions. On a Tuesday it told a customer their non-refundable fare was fully refundable. Support honoured it. Then it said the same thing to 340 more people.

The review meeting lasted twenty minutes and ended with "the model hallucinated, we'll add a guardrail". Six weeks later, a different wrong answer, same shape, same meeting. The problem was not the model. It was the postmortem.

Short version

• A postmortem is a written review after an incident, focused on causes and fixes, not blame.
• "The model did it" is a description of the symptom, not a root cause.
• The real causes are usually in the inputs, the prompt, the retrieval, the guardrails, or the evals.
• Eval gap analysis asks one question: why did our tests pass when this case exists?
• Every AI postmortem must end with a new test case, not just a new instruction in the prompt.

---

Why "the model did it" fails as a root cause

A root cause is something you can change. "The model produced an incorrect answer" is not actionable — that is the class of thing models do. It is like closing a web outage with "the server returned a 500".

Ask instead: what let that answer reach a user? There are always several layers, and each one is a place you could have stopped it.

In the travel example, at least four layers failed. Retrieval pulled an outdated document. The prompt had no instruction to cite the policy version. No guardrail compared the answer to the booking's fare rules. And the answer appeared in confident plain text with no source link.

Fixing only the model gets you one of seven. That is why the same incident came back.

The template

Copy this into your incident document.

1. Summary

Three sentences maximum. What the feature said, who saw it, and what it cost.

On 12 May, the refund assistant told customers that non-refundable fares were refundable. 341 users saw the incorrect answer over 4 days. 96 refunds were honoured, costing about 41,000 EUR.

2. Impact

• Users affected, and how you counted them
• Money, time, or trust lost
• Anything irreversible — payments sent, emails delivered, data deleted
• Whether any regulated or safety-relevant advice was given

Be exact where you can, and mark estimates as estimates.

3. Timeline

Use UTC. One line per event. Include when the change went out, not just when you noticed.

• 08 May 14:20 — Prompt v9 deployed, adds "be concise and confident"
• 08 May 14:20 — Policy documents re-indexed; 2024 policy file failed to load, no alert
• 12 May 09:12 — First incorrect answer to [email protected]
• 16 May 11:40 — Support notices a pattern, opens ticket SUP-4471
• 16 May 13:05 — Feature disabled
• 16 May 17:30 — Correct policy re-indexed, feature re-enabled with a manual check

The gap between 12 May and 16 May is often the most important thing in the whole document.

4. The example, in full

Paste the actual exchange. Never paraphrase.

• The user's exact message
• The retrieved context, or its identifiers
• The full model output
• The model name and version, temperature, and prompt version
• A request or trace ID

Without this, nobody can reproduce the incident, and an incident you cannot reproduce cannot be closed honestly.

5. What should have happened

Write the correct answer. Then say who decides what "correct" means for this feature. For a refund question, the airline's fare rules decide. For a medical question, a clinician does. Naming the authority stops the meeting drifting into opinion.

6. Cause analysis by layer

Walk the seven layers in the table above. For each, write held or failed, and one sentence.

• Input: held. The question was clear.
• Context: failed. Retrieval returned refund-policy-2023.md. The 2024 file was missing from the index.
• Prompt: failed. Prompt v9 removed the instruction to state the policy version and to say "I am not sure" when the policy is unclear.
• Model: held. No version change in the window.
• Guardrail: failed. No check comparing the answer to the fare rules on the booking record.
• Surface: failed. The answer had no source link, so support could not check it.
• Feedback: failed. No alert on refund-related answers. Found by a human after 4 days.

This section replaces "the model did it". Five failed layers means five candidate fixes, ranked by cost.

7. Eval gap analysis

Evals are the test cases you run against an AI feature before release. This section explains why they passed.

Answer four questions:

1. Did an eval cover this case? If not, why was it out of scope?
2. If one existed, did it pass? If it passed on a broken feature, the eval is wrong, not the feature.
3. Was the eval data realistic? Test sets written by the team tend to use clear, well-spelled questions. Real users do not.
4. When did the eval set last change? A test set that has not moved in six months is a snapshot of an old product.

The common finding is a shape mismatch. The eval set had "can I get a refund?" but the real user wrote "flight got moved to 6am, can i get money back or no". Different phrasing, different retrieval, different answer.

Add three things to the eval set from every incident:

• The exact failing case
• Two variations of it, worded differently
• One case that is the opposite, to check you have not over-corrected into refusing everything

8. Detection

How long between the first bad answer and someone knowing? Then ask what would have made it minutes instead of days. Usually one of:

• A rule that flags answers containing money terms for sampling
• A thumbs-down button with a required one-line reason
• A daily sample of 50 answers reviewed by a human
• An automatic check comparing answers against the source record

9. Actions

Every action needs an owner and a date. Split them into three groups.

• Stop the bleeding — disable refund answers until the guardrail ships. Owner: Priya. Done 16 May.
• Prevent this cause — alert when the policy index misses any expected file. Owner: Tomas. Due 23 May.
• Prevent the class — every money-related answer must cite a source record. Owner: Priya. Due 06 Jun.

The third one is the one that pays. Prompt tweaks belong in row two and rarely prevent a class of failure on their own.

10. What we are not doing

Write down the fixes you considered and rejected, with the reason. "We chose not to add a human review step for all answers, because volume is 12,000 a day and the sampled review covers the risky subset." Future readers will ask, and this saves the argument.

Two habits that make these reviews work

Blameless, but specific. No names attached to mistakes. Systems attached to mistakes, always. "The prompt change removed the hedging instruction" is specific and blameless at the same time.

One incident, one document. Do not merge three similar incidents into a themed review. The details are the value, and merging deletes them.

When testers find a bad AI answer in the product, the report needs the same specificity the postmortem will need later — the exact input, the full output, the page state, and the request that produced it. Capturing the console logs and network requests at that moment with a tool like Crosscheck means the trace ID is already attached when the review starts, instead of being reconstructed from memory a week later.

A checklist before you close the document

• The exact failing input and output are pasted in, not summarised
• Each of the seven layers is marked held or failed
• The eval set has grown by at least three cases
• Every action has a name and a date
• Someone outside the team could reproduce the incident from this document alone
• The words "the model hallucinated" appear nowhere as a cause

Frequently asked questions

Why is "the model hallucinated" not a root cause?

Because it describes what happened, not what allowed it to reach a user. Models will produce wrong output sometimes; the causes worth fixing are in your retrieval, prompt, guardrails, surface, and detection.

What is an eval gap?

It is the difference between the cases your test set covers and the cases real users produce. Most AI incidents live in that gap, which is why every postmortem should end with new eval cases.

Should we run a postmortem for every wrong answer?

No. Run one when the answer reached a user and caused harm, cost money, or would embarrass you if published. Track the rest as quality metrics and review them in batches.

How long should an AI postmortem take?

Aim for the document within three working days while details are fresh, and a 45-minute review meeting. If it takes longer, you are probably missing the trace data you need.

Who should attend the review?

Whoever changed the prompt, the retrieval, or the model; whoever owns the evals; someone from support who saw the user impact; and one person with no stake in the feature to ask obvious questions.