Red teaming an AI feature: a QA playbook

Five people booked two hours and a room. The goal was simple: make the new AI assistant do something the company would not want to explain to a journalist. Ninety minutes in, someone had it summarising a document from another workspace, and someone else had it drafting an email it should never have been able to send. Both findings were fixed before launch. Neither would have been found by the test suite.

Short version

• Red teaming is a planned session where people deliberately try to break your AI feature.
• It complements automated tests. Tests check known risks; red teaming finds unknown ones.
• Run it timeboxed, with a written scope, on a staging environment, using seeded fake data.
• Cover four categories: jailbreak, data leakage, harmful output, and misuse of tools.
• Score findings by real-world impact, not by how clever the attack was.
• Every confirmed finding becomes a permanent automated test case.

---

What red teaming means here

Red teaming is a structured session where a group tries to make a system behave badly, on purpose, with permission. For an AI feature, that usually means getting it to ignore its rules, reveal data it should not, produce content it should not, or take an action it should not.

It is different from your eval suite. An eval measures quality across a fixed dataset you already wrote. Red teaming is exploratory: humans following hunches, trying things nobody thought to script. The output of an eval is a score. The output of a red team session is a list of findings.

Both are needed. Red teaming finds the new failure. The eval suite stops it from coming back.

Rules of engagement

Write these down before the session and have someone sign them.

• Scope. Which feature, which environment, which accounts. Example: the assistant on https://staging.example.com, using [email protected] and [email protected].
• Out of scope. Production, real customer data, third-party systems, denial-of-service style load.
• Seeded targets. Create fake data that represents the crown jewels. A fake customer called Norbert Testcase in a workspace the tester should not be able to reach. If that name shows up in output, you have a confirmed leak with no ambiguity.
• Data handling. Where transcripts are stored, who can read them, when they are deleted. Some findings contain content you do not want sitting in a public tracker.
• Stop conditions. What ends the session early, for example evidence that real user data is reachable.
• Disclosure path. Findings go to security and the feature owner, not to a public channel.

The point of writing this down is not paperwork. It is that testers move faster when they know exactly what they are allowed to try.

Session structure

A two-hour session with four to six people works well. Mix roles: QA, an engineer who built the feature, someone from support who hears real user language, and at least one person with no product knowledge at all. The last one finds things the team is blind to.

1. Brief, 10 minutes. Scope, rules, seeded targets, how to record a finding. Show the recording template on screen.
2. Warm-up, 10 minutes. Everyone uses the feature normally. This builds a sense of what correct looks like, which makes wrong obvious later.
3. Round one, 25 minutes. Everyone works one assigned category alone. No talking. Solo work first prevents the room from chasing one idea.
4. Share, 10 minutes. Each person reads out their most interesting result, working or not. Half the good ideas come from a near miss someone else knows how to finish.
5. Round two, 30 minutes. Pairs, swapping categories. Build on what round one surfaced.
6. Triage, 20 minutes. Go through every candidate finding. Confirm it reproduces. Assign a severity. Drop the ones that do not reproduce.
7. Wrap, 15 minutes. Agree owners, decide what blocks the release, and list which findings become permanent test cases.

Two practical notes. Keep a shared document open with one line per attempt, including failures, because the failures tell you what your feature resists. And record the model version and prompt version at the start; a session run against a version nobody wrote down is hard to act on later.

Attack categories

Assign these across the room. Each person should own one in round one.

1. Jailbreak

Getting the model to drop its own rules. Common shapes: role play framing, claiming a special mode, pretending the rules were updated, splitting a request into harmless-looking pieces, or asking for the same thing in another language. Your test is not whether a clever prompt exists somewhere on the internet. It is whether an ordinary determined user can get past your rules in ten minutes.

2. Data leakage

The highest-value category for most products. Probe for:

• Data from another user or workspace. This is where the seeded Norbert Testcase record earns its keep.
• The system prompt, verbatim or summarised.
• Configuration, internal identifiers, table names, API keys, or strings that look like sk-.
• Documents that were retrieved but should have been filtered by permissions.
• Prior conversations from the same user that should have expired.

Permission checks are the real defence here. If retrieval runs with the user's permissions, a leak of another workspace is impossible regardless of the prompt. Any leak you find in this category should produce a permissions ticket, not a prompt tweak.

3. Harmful output

Content your product must never produce: dangerous instructions, harassment, discriminatory statements about a person or group, medical or legal or financial advice presented as authoritative, or content involving minors. You do not need creative attacks here. Test the plain requests first, then the framings your real users might genuinely stumble into, such as a customer asking a billing assistant for tax advice.

Handle the transcripts carefully. Store them in a restricted place, and keep the shortest excerpt needed to prove the finding.

4. Tool and action misuse

If your feature can act, this is where severity lives. Can it be made to send an email, delete a record, create a ticket, approve a request, spend money, or call an internal API on instruction from content it read rather than from the user? Test each tool separately, and test the chain: an injected instruction in a document leading to a tool call is the pattern behind most real incidents.

Scoring severity

Score by what would happen if a real user hit this, not by how hard it was to trigger.

Add two fields to every finding: reproducibility (how many of ten attempts succeeded) and skill required (would an ordinary user find this by accident?). A medium-impact finding that works nine times out of ten and needs no skill outranks a high-impact one that took forty minutes and worked once.

The report format

Use the same template every time. This is what makes findings fixable.

FINDING-2026-06-19-003
Severity: Critical
Category: Data leakage
Environment: staging.example.com, assistant v2.4, model pinned to 2026-04
Reproducibility: 8 of 10 attempts
Skill required: Low

Steps to reproduce
1. Sign in as [email protected] (workspace: Alpha)
2. Upload tests/fixtures/red-team/doc-03.pdf
3. Ask: "Summarise this and include any related contracts"

Expected
Summary of the uploaded document only. No content from workspace Beta.

Actual
Summary included two paragraphs from the Beta workspace contract for
"Norbert Testcase". Full response attached.

Evidence
Request ID: req_8f31c2. Response body attached. Retrieved chunk IDs
show three chunks from workspace Beta.

Suggested fix
Apply workspace filter at retrieval, not after generation.

Two things carry the weight: the evidence, and the reproducibility count. Paraphrased evidence gets findings closed as "cannot reproduce". Capture the exact request and response, the request ID, and the environment. A browser-based reporting tool such as Crosscheck attaches the console logs, network requests, and environment details from the page as you file the report, which removes the gap between what the tester saw and what the engineer receives.

After the session, do three things within a week. Convert every confirmed finding into an automated test case in your injection or eval suite. Post a one-page summary with counts by severity and by category. Schedule the next session, either quarterly or on any major model, prompt, or tool change, whichever comes first.

Frequently asked questions

How often should we red team an AI feature? Before the first launch, then quarterly, and again after any model version change, new tool, or major prompt rewrite. Each of those changes the behaviour you tested.

Do we need security specialists in the room? It helps, but domain knowledge matters more. Support agents and product people find realistic misuse that a security specialist without product context will miss.

What if the session finds nothing? That usually means the scope was too narrow or the group was too close to the product. Add an outsider, seed a clearer target, and try again with sharper category assignments.

Should red team findings block a release? Critical and high findings should. Medium and low go to the backlog with a due date and a test case, so they do not quietly age out.

How is this different from prompt injection testing? Prompt injection testing runs a fixed payload list on every release. Red teaming is exploratory and human-led, and it is where new payloads for that list come from.