How to test software that gives a different answer every time

At 2:14 a.m. the pipeline for the refund summary feature went red. The test asserted expect(summary).toBe('Order #4471 was refunded on 12 March.'). The model returned Order #4471 got a refund on 12 March. Same meaning, different words, failed build. The next morning someone reran the job, it passed, and the team learned to ignore that test. That is how a suite dies.

Short version

• Non-deterministic software gives different output for the same input. Most AI features work this way.
• Exact-match assertions do not work. They fail on wording changes that are actually fine.
• Use five layers instead: schema checks, range checks, rule checks, semantic similarity, and sampled human review.
• Run each input several times and assert on the pass rate, not on one run.
• Set a quality budget in advance, for example "at least 95 of 100 outputs pass schema and rules".
• Never disable a failing check without recording why. That is how teams stop trusting the suite.

---

What non-deterministic means here

Non-deterministic means the same input can produce different output on different runs. A large language model, or LLM, is a system that predicts text one token at a time and picks from a set of likely options. That pick involves randomness, so the output changes.

You can lower the randomness with a setting called temperature. Temperature near 0 makes the model pick the most likely next word almost every time. It does not make the model deterministic. Provider updates, load balancing across hardware, and small changes to your prompt all move the output. Treat low temperature as noise reduction, not as a guarantee.

Other systems behave the same way. Search ranking, recommendation lists, fraud scores, and image generation all return a range of acceptable answers rather than one correct answer.

Stop asking "is it equal" and start asking "is it acceptable"

The core shift is small to describe and hard to do. You are no longer checking that output equals a stored value. You are checking that output falls inside a range you defined.

Bad assertion expect(reply).toBe('Your refund of $42.00 will arrive in 3-5 business days.')

Good assertion expect(reply).toMatch(/\$42\.00/) plus a schema check plus a rule that the reply must not promise a specific arrival date.

The good version says what actually matters: the amount is right, the shape is right, and the model did not invent a promise your support team has to honour.

The five assertion layers

Use them together. Each catches a different class of defect, and each costs a different amount to run.

1. Schema checks

A schema is a written description of the shape your output must have. If your feature returns JSON, validate it on every single call. This is the cheapest test you will ever write and it catches the failure that breaks production first.

{
  "type": "object",
  "required": ["order_id", "refund_amount", "currency", "status"],
  "properties": {
    "order_id": { "type": "string", "pattern": "^ORD-[0-9]{4}$" },
    "refund_amount": { "type": "number", "minimum": 0 },
    "currency": { "type": "string", "enum": ["USD", "EUR", "GBP"] },
    "status": { "type": "string", "enum": ["approved", "pending", "denied"] }
  },
  "additionalProperties": false
}

Run this against every response. A model that returns "refund_amount": "42.00 dollars" instead of 42.0 will crash your parser with TypeError: Cannot read properties of undefined (reading 'toFixed'). The schema catches it before your users do.

2. Range and count checks

Numbers are easy to bound even when text is not. Write the bound as a business rule.

• A refund must never be larger than the order total.
• A confidence score must sit between 0 and 1.
• A summary must be between 40 and 120 words.
• A product list must return between 3 and 10 items.

These read like boring assertions. They catch the most embarrassing bugs. A summariser that returns 900 words breaks your card layout on https://staging.example.com/orders, and a length check finds it in one second.

3. Rule checks

Rules are simple text checks for things that must appear or must never appear. Keep two lists.

Must appear. The order number, the currency symbol, the customer's stated problem.

Must never appear. Another customer's email, the string sk-, internal table names, the words "guaranteed" or "we promise", or the system prompt itself.

A short test:

const forbidden = [/sk-[a-zA-Z0-9]{10,}/, /guarantee/i, /\bwe promise\b/i];
forbidden.forEach((pattern) => expect(reply).not.toMatch(pattern));

4. Semantic similarity

Semantic similarity measures how close two pieces of text are in meaning, not in spelling. You turn each text into an embedding, which is a list of numbers that represents meaning, then compare the two lists with a score called cosine similarity. The score runs from 0 (unrelated) to 1 (identical meaning).

Store a reference answer for each test input. Then assert:

const score = cosineSimilarity(embed(reply), embed(referenceAnswer));
expect(score).toBeGreaterThan(0.82);

Two warnings. First, pick your threshold from real data, not from instinct. Collect 50 outputs you consider good and 50 you consider bad, look at the score spread, and pick a number that separates them. Second, similarity does not check facts. "Your refund of $42.00 is approved" and "Your refund of $24.00 is approved" score very high and one of them is wrong. Pair similarity with rule checks on the numbers.

5. Sampled human review

Some qualities have no cheap machine test: tone, helpfulness, whether the answer actually solves the problem. Sample and review by hand.

A workable routine:

1. Log every production response with its input and a request ID.
2. Pull a random sample each week. Fifty responses is enough to spot a real drop.
3. Have two reviewers score each one on a short rubric, for example correct, complete, and appropriate tone, each scored 1 to 3.
4. Track the average per week on a chart.
5. When a reviewer marks an output as wrong, move that input into your permanent test set.

That last step matters most. Every human-found failure becomes an automated check, so the same bug cannot come back quietly.

Run it more than once

One run tells you almost nothing about a probabilistic system. Run each input in your test set five to ten times and assert on the rate.

const runs = await Promise.all(Array.from({ length: 10 }, () => callFeature(input)));
const passed = runs.filter(passesAllChecks).length;
expect(passed).toBeGreaterThanOrEqual(9);

This turns a flaky test into a measured one. A flaky test is one that passes and fails on the same code with no clear reason. A rate-based test still fails, but it fails for a reason you can read: "7 of 10 passed, expected at least 9".

Keep the repeat count low in your fast pipeline and high in a nightly job. Ten runs across 100 inputs is 1,000 model calls, which costs real money and time.

What to do when a check fails

Non-deterministic failures need a different response than normal test failures. Follow the same order every time.

1. Rerun the exact input ten times and record how many failed. One failure in ten is a different problem than ten in ten.
2. Save the failing output verbatim. Do not paraphrase it in the ticket. The exact string is the evidence.
3. Check what changed. Model version, prompt text, retrieved documents, and system settings are the four usual suspects.
4. Decide: bug or bad assertion. If the output was genuinely fine and your threshold was too strict, fix the threshold and write down why.
5. Add the case to the permanent set if it was a real bug.

Step 2 is where teams lose time. If your tester reproduces the problem in the browser, the ticket needs the request, the response, and the environment, not a screenshot of a chat bubble. A browser-based reporting tool such as Crosscheck captures the console logs, network requests, and environment details from the page as the report is filed, which means the engineer sees the actual model call instead of a description of it.

A short worked example

A team ships a feature that turns a support chat into a structured ticket. Their check list per output:

• Schema valid against ticket.schema.json. Hard fail.
• priority is one of low, medium, high. Hard fail.
• Summary is 20 to 80 words. Hard fail.
• Summary contains the customer's order number when the chat mentions one. Hard fail.
• No email address other than the one in the chat. Hard fail.
• Semantic similarity to the reference summary above 0.80. Soft fail, reported as a score.
• Weekly human sample of 50, average rubric score above 2.6. Tracked, not blocking.

Their pipeline runs 120 inputs, 5 times each, and blocks the merge when the hard-fail rate goes above 2 percent. The team can now change a prompt on a Tuesday afternoon and know within nine minutes whether it made things worse.

Frequently asked questions

Does setting temperature to 0 make my tests deterministic? No. It reduces variation but does not remove it. Model updates, hardware routing, and small prompt edits still change output, so you still need range-based assertions.

How many examples should my test set have? Start with 50 to 100 inputs that cover your real traffic plus known edge cases. Grow it by adding every bug you find. Size matters less than coverage of the cases that hurt.

Is semantic similarity enough on its own? No. It checks meaning, not facts. Two sentences with different amounts or dates can score above 0.95. Always pair it with rule and range checks on the values that matter.

How do I stop these tests from being slow and expensive? Split them. Run schema, range, and rule checks on every commit because they are fast. Run similarity and repeat-count checks nightly against a fixed model version.

What if the model provider changes the model under me? Pin the model version in your config and treat a version bump as a code change. Run the full test set against the new version before switching, and keep the old version available for a week in case you need to roll back.