Where AI agents belong in a CI pipeline

A team at a payments company added an AI reviewer to their merge gate. It blocked a one-line config fix at 11pm during an incident, because it "could not verify the change was safe". The on-call engineer spent twenty minutes finding the admin override.

The tool was not bad. The slot was wrong. AI agents earn their place in CI when they advise, and they cause damage when they decide.

Short version

• CI (continuous integration) is the automated pipeline that builds and tests every code change.
• Put AI agents in advisory slots: comments, summaries, and triage suggestions.
• Three jobs pay off fast — non-blocking review, flake triage, and pull request risk summaries.
• Never let an agent be the only thing standing between a change and production.
• Measure the agent like a test: how often is it right, and what does a wrong answer cost.

---

What an "AI agent" means here

An AI agent in CI is a program that reads your code, test output, and history, then writes something useful back. It might post a comment on a pull request, label a failing test, or open a ticket.

It is different from a linter. A linter follows fixed rules and gives the same answer every time. An agent uses a language model, so the same input can produce a slightly different answer on two runs. That single difference decides where it belongs.

Anything that must be repeatable — a security scan, a schema check, a license check — should stay a deterministic tool. Anything that is judgment, and where a human reads the output, is fair game for an agent.

The three jobs that pay off

1. Non-blocking review comments

Let the agent read the diff and post comments. Do not let it approve or reject. Humans still review; the agent just gets there first and catches the boring things.

Good comments look like this:

src/checkout/total.ts:44 — discount can be undefined when the promo code lookup misses. Line 51 calls discount.toFixed(2), which will throw TypeError: Cannot read properties of undefined (reading 'toFixed').

Bad comments look like this:

Consider refactoring this function for improved readability and maintainability.

The first one names a file, a line, a condition, and the exact error. The second one is noise. If your agent produces mostly the second kind, tune the prompt or turn it off.

Set a hard cap: no more than five comments per pull request. An agent that leaves thirty comments trains everyone to ignore all thirty.

2. Flake triage

A flaky test is one that passes and fails on the same code, with no change in between. Most teams have a folder full of them and no time to sort them out.

This is a strong fit for an agent, because the work is reading — logs, screenshots, timing data, and the last fifty runs of the same test.

A useful triage output:

checkout.spec.ts > applies promo code failed 6 of the last 40 runs on main. All 6 failures show TimeoutError: locator.click: Timeout 5000ms exceeded on the #apply-promo button. All 6 ran on the ci-runner-4 agent. Likely infrastructure, not product. Suggested owner: platform team.

That is a real hypothesis with evidence behind it. A human confirms it in two minutes instead of thirty.

Important: the agent proposes, a human quarantines. Do not give the agent permission to disable tests on its own. A tool that can silence failing tests will eventually silence a real one.

3. Pull request risk summaries

Reviewers open a pull request and see 40 changed files. The agent's job is to say where to look first.

A risk summary should answer three questions:

1. What did this change actually do, in two sentences?
2. Which parts touch money, authentication, data deletion, or public APIs?
3. What is missing — tests, migrations, feature flags, rollback notes?

Example:

This adds partial refunds to the orders API. Highest risk: refundOrder() now writes to payments.ledger inside the same transaction as the order update. No test covers a refund larger than the original charge. The migration 0142_add_refund_amount.sql has no down migration.

Reviewers still review. They just start in the right place.

Where NOT to put them

The rule: an agent should never be the last thing between a change and users.

The pattern behind the "no" column: the action is either irreversible, or it removes a human, or it needs to be identical every time it runs.

There is one more reason to keep agents out of blocking gates. Model providers have outages, rate limits, and version changes. If the agent is a required check, an upstream incident becomes your incident. Advisory checks fail quietly. Blocking checks fail loudly and stop your team.

How to roll one out without regret

1. Pick one job. Start with pull request risk summaries. It is the lowest risk and the easiest to judge.
2. Run it in shadow mode for two weeks. Post output to a private Slack channel, not the pull request. Nobody is affected while you learn.
3. Score 30 real samples. For each one, mark it useful, harmless, or wrong. Keep the sheet.
4. Set your bar before you look. For example: at least 60 percent useful and under 10 percent wrong before it goes public.
5. Turn it on as a comment only. Never required in your branch protection rules.
6. Add a mute button. One label, like no-ai-review, that skips the agent on a pull request. People will need it during incidents.
7. Re-score every quarter. Models change under you. So does your codebase.

Step 3 is the one teams skip, and it is the one that matters. Without a scored sample you are arguing about vibes.

What to measure once it is live

Track three numbers, not ten:

• Useful rate — how many agent comments led to a change in the code. Ask reviewers to react with a thumbs-up. Rough is fine.
• Noise rate — comments marked resolved with no code change. Above 50 percent and people stop reading.
• Cost per pull request — the actual model spend. Divide the monthly bill by the number of pull requests. If it costs more per review than a person's time saved, stop.

Watch for the failure mode where the useful rate looks fine but reviewers have quietly started collapsing the agent's comment without reading it. Ask them directly in a retro. The dashboard will not tell you.

The human bugs are still yours

An agent reading a diff cannot see what a user sees. It does not know that the promo banner overlaps the checkout button on a 1280px screen, or that the network request to https://staging.example.com/api/promo returns a 500 for accounts created before 2024.

That evidence comes from a person using the product and capturing what happened. Tools like Crosscheck handle that side — a tester reports the bug from the page and the screenshot, console errors, network calls, and environment details come along automatically.

CI agents and human reports cover different halves of the same problem. Neither replaces the other.

A quick sanity check before you add any agent

Ask these four questions. If any answer worries you, the slot is wrong.

• If the agent is wrong, who finds out, and how fast?
• If the model provider goes down for an hour, what breaks?
• Can a human override it in under 30 seconds at 2am?
• Would I be comfortable if this ran on the change that fixes a live outage?

The last one catches most bad ideas. Incidents are when your pipeline is under the most pressure and your people have the least patience.

Frequently asked questions

Can an AI agent replace code review?

No. It can make review faster by pointing at the risky parts of a diff, but a human still needs to decide whether the change is right for the product. Treat agent comments as a first pass, not a verdict.

Should the AI agent be a required status check in GitHub?

No. Required checks must be reliable and repeatable. A language model is neither, and an outage at the provider would block every merge in your organisation. Keep it as a comment-only check.

How do I stop the agent from leaving too many comments?

Cap it in the prompt and in the code — for example, at most five comments, ranked by severity. Also drop any comment that does not name a specific file and line. Vague style advice is the main source of noise.

Is it safe to send our private code to a model provider?

That depends on your contract and your data rules. Check whether the provider trains on your input, where the data is stored, and how long they keep it. Some teams only run agents on public repositories or use a self-hosted model for private ones.

What is the cheapest first agent to try?

A pull request risk summary. It runs once per pull request, uses a small amount of input, and its value is easy to judge — either reviewers found it helpful or they did not.