Testing voice agents

A support line for a bank went live with a voice agent. It worked in every demo. On day one, a caller said "no, no, that's not my account" while the agent was still talking. The agent finished its sentence, missed the interruption, and confirmed a transfer to the wrong account.

Nothing in the test plan covered someone talking over the bot. That is the pattern with voice: the failures are not in the words, they are in the timing, the noise, and the pauses.

Short version

• A voice agent is software that listens to speech, decides what to do, and speaks back.
• Test the pipeline in three parts: hearing, deciding, speaking. Failures look different at each stage.
• Barge-in — the caller talking over the agent — is the most under-tested behaviour.
• Silence is a signal, not an error. Decide what each length of pause means.
• Set a transcription error budget before launch, and measure against real recordings, not clean studio audio.

---

The three stages, and where each one breaks

Every voice agent runs the same loop:

1. Hearing — speech becomes text. This is ASR (automatic speech recognition), sometimes called speech-to-text.
2. Deciding — the text becomes an intent and an action. Usually a language model or a rules engine.
3. Speaking — the response becomes audio. This is TTS (text-to-speech).

Bugs in stage 1 look like nonsense answers. Bugs in stage 2 look like the agent understood but chose wrong. Bugs in stage 3 look like the agent said the right thing badly — wrong pronunciation, wrong pace, wrong pause.

When you file a bug, always say which stage. "The agent got it wrong" is not enough. Include the transcript the system produced, because that single line usually tells you the stage.

Bad report: The bot did not understand my address.

Good report: I said "42 Whitfield Road". The transcript shows "40 to Whitfield Road". The agent then asked for a valid house number. Stage 1 failure, ASR.

Barge-in

Barge-in is when the caller speaks while the agent is still speaking. Real people do it constantly, especially when the agent is reading a long list.

There are three behaviours a system can have:

• No barge-in — the agent ignores input until it finishes. Frustrating, but predictable.
• Full barge-in — any sound stops the agent immediately. Prone to false stops from a cough or a passing car.
• Smart barge-in — the agent stops on speech but not on noise, usually with a short delay.

Test cases to run:

1. Interrupt in the first second of the agent's turn.
2. Interrupt in the last second.
3. Interrupt with a single word: "stop".
4. Interrupt with background noise only — a door closing, a dog.
5. Say "mm-hmm" mid-sentence. The agent should not treat a backchannel as a command.
6. Interrupt during a confirmation of a money transfer, then say "no".
7. Interrupt twice in a row, fast.

Case 6 is the one from the bank story. Any turn where the agent is about to do something irreversible needs a barge-in test of its own.

Record the time from your speech starting to the agent's audio stopping. Anything above roughly half a second feels like the agent is ignoring you. Write down the number you measured, not "it felt fine".

Accents, speed, and voices

Most voice test plans use a handful of colleagues who all work in the same office and often share an accent. That is the narrowest possible sample.

Build a coverage matrix and treat gaps as risk, not as a nice-to-have.

• Accent or region — the five largest groups in your user base
• Age — one older speaker, and one child if children use it
• Speaking speed — fast, normal, and slow
• Voice pitch — low and high
• Speech differences — a stutter, a lisp, a speaker using a wheelchair-mounted mic
• Background — quiet room, car, street, cafe, speakerphone
• Bilingual — names and words from another language mid-sentence

The bilingual line catches a lot. A user saying "my name is Ngozi Adeyemi" or "I want to pay my Grünwald invoice" will break systems tuned on one language.

For each combination, run the same five utterances. Then compare accuracy across rows. If one accent is 15 points worse than the best row, that is a product defect, not an edge case. It means some of your users get a worse product than others.

Silence handling

Silence carries meaning, and most agents treat all silence the same way. Decide the rules and test them.

A reasonable set:

• Under 0.8 seconds — the caller is still speaking. Do not respond.
• 0.8 to 3 seconds — turn is over. Respond.
• 3 to 8 seconds — prompt gently: "Are you still there?"
• 8 to 20 seconds — repeat the question, once.
• Over 20 seconds — offer a human, then end the call politely.

Test each boundary from both sides. The common bug is at the first one: the agent cuts in while the caller pauses to think, or to read a card number off a screen.

Also test silence with meaning. If you ask "should I cancel the order?" and the caller says nothing, the agent must not treat silence as yes. Silence is never consent. Write that as an explicit test.

Transcription error budgets

You cannot get perfect transcription, so decide in advance how much error you accept and where.

The standard measure is WER (word error rate) — the share of words the system got wrong, including words it added or dropped. A WER of 8 percent means roughly 8 words in every 100 are wrong.

But a flat WER hides the thing that matters. One wrong word in "I think that's fine" costs nothing. One wrong digit in a sort code costs money. So set the budget per field, not for the whole call.

Two rules go with the table. First, any field with a budget under 5 percent must be read back to the caller for confirmation. Second, digits should be confirmed in groups: "that's four, two, one — is that right?"

Measure the budget against recordings from real calls, with real background noise. Clean audio recorded on a good microphone in a quiet room will give you a number you cannot ship on.

Building a regression set

Screen tests can be re-run for free. Voice tests need audio, so build an audio library once and reuse it.

1. Collect 100 to 300 real call recordings, with consent and with personal data removed.
2. Write the correct transcript for each one by hand. This is slow and it is the whole asset.
3. Tag each clip: accent, noise level, intent, and whether it contains digits.
4. Run every release against the full set. Compare WER per tag, not overall.
5. Add every production failure to the set as a new clip, the same way you would add a regression test.
6. Re-measure whenever you change a model, a prompt, or a phone provider.

Step 6 matters more than it sounds. Changing an audio codec or a telephony provider can move accuracy as much as changing the model.

Reporting voice bugs so they get fixed

A voice bug report needs more than steps. Include:

• The audio clip, or a call ID and timestamp
• The transcript the system produced
• What you actually said, written out
• The intent the agent chose, if your logs expose it
• The stage you think failed: hearing, deciding, or speaking
• Environment — phone or web, network, headset or speakerphone, codec if known

If part of your product is a web-based voice widget, the browser side matters too. A microphone permission that was silently denied, or a console error like DOMException: Permission denied, explains a "the agent never heard me" report in seconds. A tool such as Crosscheck captures the console logs, network requests, and environment details from the page at the moment the tester reports the problem, which saves a round trip.

Frequently asked questions

What is barge-in and why does it matter so much?

Barge-in is the caller speaking while the agent is still talking. It matters because it is how real people correct mistakes, and an agent that ignores interruptions will complete actions the caller was trying to stop.

How many accents do we need to test?

Cover the largest groups in your actual user base, then add any group where you have complaints. The goal is not full coverage of every accent, it is making sure no group gets a clearly worse experience than another.

What is a good word error rate to aim for?

There is no single good number, because it depends on the field. Casual conversation can tolerate around 15 percent, while digits like account numbers need to be under 1 percent and confirmed back to the caller.

Can we test voice agents automatically?

Partly. You can replay recorded audio against the system and compare transcripts and chosen intents automatically. Judging whether a response sounded natural or polite still needs people listening.

Should silence ever count as agreement?

No. Treat silence as "no answer yet" and prompt again. Any flow that takes silence as confirmation of a payment, a cancellation, or a data change will eventually harm a user.