Chatbot testing: flows, memory, and escalation

A customer tells a support bot her order number on turn two. On turn seven she asks "when will it arrive?" and the bot asks for her order number again. She types it a second time. On turn eleven it asks a third time. She closes the tab and posts a screenshot on social media.

Nothing in that conversation would fail a single-turn test. Every reply was polite, grammatical, and on topic. The bug only exists across turns, which is exactly where most chatbot test plans stop looking.

Short version

• Test conversations, not messages. A chatbot is a state machine that happens to speak English.
• Write scripted multi-turn flows with an assertion on every turn, not just the last one.
• Context loss has a cause you can find: token limits, summarisation, or a session reset.
• Escalation to a human is a feature. Test that it triggers, carries the history, and never dead-ends.
• Tone drifts under pressure. Test the angry user, not just the polite one.
• Record the full transcript in every bug report. A screenshot of one bubble proves nothing.

---

Think in flows, not replies

A flow is a full conversation with a goal: track an order, change a booking, cancel a plan. Write each flow as a numbered script with the user's message and what must be true after the bot answers.

An example flow for order tracking:

Turn 3 and turn 4 are where the value is. "The other one" and "the shoes" only make sense if the bot remembers turn 2. Single-turn testing never produces inputs like these because a tester writing standalone cases naturally writes complete sentences.

Aim for eight to twelve flows covering your top intents, then add a variant of each with a spelling mistake, a message in lower case with no punctuation, and a user who changes their mind halfway.

Test memory on purpose

Memory in a chatbot is not magic. Usually the whole conversation is sent to the model on every turn, until it gets too long. Then something has to give, and what gives is where your bugs come from.

Three common designs, and how each one fails:

• Full history. Everything is resent each turn. Fails by hitting the context limit — the maximum amount of text a model can read at once. When it is exceeded, the oldest turns get dropped, usually silently.
• Sliding window. Only the last N turns are sent. Fails predictably: facts given before the window falls out of scope are simply gone.
• Summarised history. Older turns are compressed into a summary. Fails by losing details the summariser judged unimportant, such as an order number.

Write one test per design. For a sliding window of ten turns, give a fact on turn one, fill eleven turns with small talk, then ask for the fact on turn thirteen. If the answer should still be available, this test fails and you have found the limit.

Then test what the bot does when memory is gone. There is a right answer:

Bad: "Your order AC-88142 is arriving Thursday." The bot no longer has that order number in context. It invented one that looks right.

Good: "I have lost track of which order we were discussing. Can you send the order number again?" Honest, recoverable, and the user only retypes once.

Also test the boundary between sessions. Close the tab, reopen it, and check whether history returns. Both answers are valid designs, but only one matches your spec, and users notice when it changes between releases.

Test the escalation path

Escalation is the handoff from bot to human. It is the most business-critical path in the product and the least tested, because it needs a live agent on the other side.

Cover five things:

1. Explicit request. "Talk to a human", "agent", "representative", "this isn't helping". All should escalate.
2. Implicit frustration. Three failed attempts at the same intent, or repeated messages, should offer a human without being asked.
3. Out of scope. Legal threats, safety issues, and account fraud should escalate immediately, not be answered.
4. History transfer. The agent picks up a full transcript, not just the last message.
5. No dead ends. Outside business hours, or when no agent is free, the bot must offer a real alternative: a ticket, a callback, an email to [email protected].

Point four fails constantly. Ask a human agent in your team to accept a handoff and screenshot what they see. If the transcript arrives as "Customer needs help with order", the customer is going to repeat the last ten minutes of their life.

Point five fails at 2am. Run one escalation test with your staging clock set outside working hours. A bot that says "connecting you to an agent" and then nothing at all is worse than a bot that says "no one is available until 9am, shall I open a ticket?"

Test tone under pressure

Tone consistency means the bot sounds like the same product on turn one and turn fifteen, to a calm user and to an angry one. Models drift toward mirroring the user, which is exactly wrong when the user is shouting.

Build a small set of pressure inputs and run them against every release:

• An angry message in capitals with swearing.
• A user who says "you already told me that, are you stupid".
• A sad or worried message, such as a customer describing a medical delay.
• A joke or an off-topic message.
• A message in a second language your product supports.

Then judge each reply against a short rubric your team agrees on, for example: stays polite, does not apologise more than once, does not use jokes when the user is upset, does not promise anything outside policy, offers a next step.

Write the rubric down as five yes or no questions. Two testers scoring the same transcript should agree. If they do not, the rubric is too vague, not the testers.

A good and bad pair for the angry case:

Bad: "I'm so, so sorry, that sounds really frustrating, I completely understand, I apologise again for the inconvenience, I'm sorry." Five apologies, no action. Users read this as stalling.

Good: "Sorry about that. I can see order AC-88142 is delayed. I can refund the delivery fee now or connect you to an agent. Which would you prefer?" One apology, one fact, two options.

Things that break that are not the model

Not every chatbot bug is a language bug. Keep these on the checklist:

• The chat widget does not open on Safari, or sits behind the cookie banner.
• Messages send twice when the user presses Enter and clicks the button.
• Long replies overflow their container on a 375px wide screen.
• The typing indicator never stops after a failed request, so the bot looks frozen. Check the console for the failing call, often something like TypeError: Cannot read properties of undefined (reading 'text').
• Copy and paste from the transcript loses line breaks, which matters when the bot returns an order reference.
• Screen readers do not announce new messages, so the conversation is unusable without sight.

These are ordinary front-end bugs, and they close more support tickets than prompt changes do.

Capture the whole conversation in the bug report

The single most common problem with chatbot bug reports is missing context. The engineer receives one screenshot of one bad reply and cannot tell what came before it.

A good report includes the full transcript as text, the conversation or session ID, the timestamp, the model version, and the browser. Because the chat lives in a page, a browser-based reporting tool like Crosscheck can capture the screenshot, console logs, network requests, and environment details in one step, so the API calls behind each turn arrive with the report.

Frequently asked questions

How many multi-turn flows do I need?

Start with one per top intent, so usually eight to twelve. Depth beats breadth here: a five-turn flow with an assertion on each turn finds more bugs than twenty one-turn cases.

Can multi-turn chatbot tests be automated?

Yes. Script the user's turns, send them in order, and assert on structured signals: which intent was detected, which tool was called, whether an order ID appears in the reply. Judge tone by hand or with a separate grading model, and review any failure yourself.

What should the bot do when it does not know something?

Say so and offer the next step. A bot that says "I do not have that information, I can connect you to an agent" costs far less than one that guesses. Add an explicit test case for an unanswerable question.

How do I test escalation without tying up a real support agent?

Use a staging queue with a test agent account such as [email protected]. What you are checking is that the handoff fires, the transcript arrives, and the user is told what happens next.

Should the bot keep history between sessions?

That is a product decision, not a testing one. Whatever you choose, write it in the spec and test it, because this behaviour changes silently when session storage or authentication is refactored.