Latency and token cost as test criteria

A team shipped a small improvement to their document summariser: three extra examples in the prompt to fix a formatting complaint. Quality scores went up. Two weeks later, finance asked why the monthly bill had gone from 4,000 to 11,000, and support asked why the page felt slow.

The change added about 1,900 tokens to every single request. Nobody tested for that, because nobody had written down what a request was allowed to cost.

Short version

• A token is a chunk of text, roughly four characters. You pay per token in and per token out.
• Give every AI feature a written budget: maximum tokens in, tokens out, and seconds to respond.
• Gate on p95, not the average. The average hides the users who leave.
• Assert token counts in CI. A prompt edit is a cost change and should be reviewed like one.
• With streaming, the number users feel is time to first token, not total time.
• Long context is a cost trap: retrieval, chat history, and few-shot examples all grow quietly.

---

Write the budget before you build

A budget is three numbers per feature, agreed by product, engineering, and testing. Without it, every performance conversation becomes an opinion.

Two things make this table useful. First, the numbers differ by feature, because a user waiting for autocomplete and a user waiting for a 40-page summary have different patience. Second, they are testable. A test can count tokens and measure milliseconds. It cannot measure "fast enough".

Derive the cost per request from the budget and multiply by expected volume. If support chat runs 30,000 times a month at the budgeted size, you know the monthly ceiling before you write any code. That single calculation prevents most unpleasant invoices.

Measure p95, not the average

p95 latency means: 95 percent of requests finish faster than this number. Five percent are slower.

Averages lie badly here, because language model response times have a long tail. A feature can average 2 seconds while one request in twenty takes 15 seconds. The average looks healthy and one user in twenty gives up.

Report four numbers for every run:

• p50 — the typical experience.
• p95 — your gate. This is what you promise.
• p99 — the tail. Useful for spotting timeouts and retries.
• Max — the worst case, which tells you whether your client timeout is set correctly.

To measure them honestly you need enough samples. Fifty requests gives a rough p95. Two hundred gives a usable one. Run them with realistic inputs, because a 200-token test prompt will never reproduce the latency of a 12,000-token production prompt.

Also measure at the right layer. Two numbers matter and they are not the same:

Model latency: time from your server sending the request to the last token arriving. User latency: time from the click to the answer being readable on the page.

User latency includes your retrieval step, any reranking, network time, and rendering. Teams often optimise the first while the second stays flat, because the real cost was a 4-second vector search.

Put token counts in CI

Token count is a deterministic property of your prompt. You can assert on it exactly, and you should.

A workable setup:

1. For each feature, build the prompt from a fixed sample input.
2. Count the tokens with the same tokenizer your provider uses.
3. Compare against a checked-in baseline file.
4. Fail the build if input tokens grow by more than 10 percent without the baseline being updated in the same pull request.
5. Print the estimated cost change in the CI output, in money.

That fifth step changes behaviour more than the gate does. A reviewer who sees "prompt grew 1,900 tokens, estimated +7,000 per month at current volume" will ask a question. A reviewer who sees a diff of three example blocks will approve it.

Output tokens are not deterministic, so treat them differently. Run the sample set a few times, take the p95 output length, and alert on a rise rather than blocking the build.

Watch these four sources of quiet growth:

• Retrieved chunks. Raising top-k from 5 to 8 adds thousands of tokens per request.
• Chat history. Each turn adds the previous ones. Test a 20-turn conversation, not a 2-turn one.
• Few-shot examples. Every example added to fix a bug stays forever.
• Tool definitions. Agent tool schemas are sent on every call. Twelve tools can be larger than the user's question.

Test the tail: timeouts, retries, and rate limits

Cost and latency tests should include failure paths, because failures are expensive twice.

• Timeout. Force a slow response and check the client gives up cleanly at the budgeted limit rather than hanging. A spinner that never stops is a bug even when the backend recovers.
• Retry storms. If your client retries three times on failure, a slow provider triples your bill and your latency. Assert that retries are capped and that they back off.
• Rate limits. Trigger a 429 Too Many Requests and confirm the queue drains instead of dropping work.
• Partial failures. A streaming response that stops halfway should show what arrived and offer a retry, not a blank box and TypeError: Cannot read properties of undefined (reading 'choices') in the console.

Run these against a staging endpoint such as https://staging.example.com/api/summarise where you can inject delays and errors deliberately.

Streaming changes what you measure

Streaming means tokens appear as they are generated instead of arriving all at once. It does not make the response faster. It makes the wait feel shorter, and it moves your key metric.

With streaming, add tests that blocking interfaces never need:

• The first token appears within budget, even when the total response is long.
• The stream does not stall for more than 3 seconds mid-answer, which reads as a freeze.
• Cancelling mid-stream stops the request server-side. If it does not, you pay for tokens nobody reads.
• Partial content is not treated as final. Test that a form does not submit a half-written JSON payload.
• The page does not jump as text arrives. Layout shift during streaming is a real usability bug.

A good and bad pair for the same 9-second response:

Bad: A spinner for 9 seconds, then the full answer. Users refresh at around 6 seconds and you pay twice.

Good: First words at 1.1 seconds, steady output, a visible stop button. Same 9 seconds, and almost nobody refreshes.

Report performance bugs with the evidence attached

"The AI feature is slow" cannot be fixed. A useful report names the number, the budget it broke, and the request behind it.

Include the request ID, the timestamp, the model version, input and output token counts, the measured latency, and the network trace for the page. Because most of this lives in the browser's network panel, a tool like Crosscheck that captures screenshots, console logs, network requests, and environment details from the page in one step saves the back and forth of asking a tester to reproduce it with DevTools open.

Frequently asked questions

Should latency tests block a release?

Gate on p95 for user-facing features and treat token count growth as a blocking check. Batch and background jobs can use alerts instead, since a slow nightly job rarely costs a customer.

How many requests do I need for a reliable p95?

At least 200 with realistic inputs. Fewer than 50 gives a p95 that swings widely between runs and will produce false alarms.

Why does my cost rise when the prompt has not changed?

Usually the input grew for another reason: more retrieved chunks, longer chat histories, larger user documents, or added tool definitions. Log input token counts per request and chart them, and the cause shows up quickly.

Does streaming reduce cost?

No. You pay for the same tokens. It improves perceived speed and lets users cancel early, which is where the savings come from if cancellation actually stops the request.

How do I estimate cost before launch?

Take the budgeted input and output tokens, apply your provider's current per-token rates, and multiply by forecast volume. Then run 200 real requests and compare, because real inputs are almost always larger than the samples used for planning.