Reviewing AI-generated tests: what to reject on sight

A pull request lands with 340 new lines of tests and a coverage badge that moved from 61% to 88%. Everything is green. It took the author four minutes to generate.

Two weeks later a null value reaches the invoice total and production shows $NaN to customers. You go back and read the tests. Every one of them passes whether the code works or not.

Generated tests are cheap, and cheap things get merged without being read. This is the checklist for reading them.

Short version

• The only question that matters: if the feature broke, would this test fail?
• Four patterns cause most of the damage: tautologies, over-mocking, coverage theatre, and unfalsifiable tests.
• Test the test. Break the code on purpose and confirm the test goes red.
• Coverage percentage measures lines run, not behaviour checked. They are not the same thing.
• Reject any test whose assertion restates the mock you just wrote.
• A smaller suite of real tests beats a large suite of decorative ones.

---

The one question

Before the detailed patterns, hold this in your head while reading any generated test:

If I introduced a realistic bug in the code under test, would this specific assertion turn red?

If the answer is no, the test is not a test. It is a piece of code that runs and reports success. Everything below is a specific way of answering no.

---

Pattern 1: tautological assertions

A tautology is a statement that is true by construction. In tests, it means the assertion checks something that cannot be false.

Reject:

const total = calculateTotal(items);
expect(total).toBe(calculateTotal(items));

Reject:

const user = { name: 'Ada', role: 'admin' };
expect(user.role).toBe('admin');

Accept:

const total = calculateTotal([{ price: 19.99, qty: 2 }, { price: 5.00, qty: 1 }]);
expect(total).toBe(44.98);

The first calls the function twice and compares it to itself. It passes even if the function returns garbage. The second asserts on a value the test itself just wrote on the line above. The third has a hand-calculated expected value that the function must actually produce.

The tell: the expected value is derived from the code under test, or from the test's own setup. Real assertions use values a human worked out from the requirements.

A softer version of the same problem is the assertion that checks almost nothing:

expect(result).toBeDefined();
expect(response.status).toBeTruthy();
expect(list.length).toBeGreaterThan(0);

These pass when the function returns an empty object, a status of 418, or a list of the wrong items. Ask for the exact value instead.

---

Pattern 2: mocked-away logic

A mock is a fake stand-in for a real dependency, used so a test can run without a database or network. Mocks are necessary. The failure is mocking the thing you claim to be testing.

Reject:

jest.mock('./pricing', () => ({ applyDiscount: () => 90 }));

test('applies the 10% discount', () => {
  expect(applyDiscount(100)).toBe(90);
});

The test asserts that the mock returns what the mock was told to return. The real discount logic is never executed. Change the discount rule to 50% in the real file and this test still passes.

The rule is simple: mock the boundary, never the behaviour. Mock the HTTP call, the clock, the payment provider, the file system. Do not mock the function whose name appears in the test title.

When you review, list every mock in the file and ask what each one removes:

One more version worth catching: a mock that never gets checked. If the test mocks the email sender and then never asserts that it was called, with what arguments, the mock is only there to stop an error. That is fine for setup, but it is not coverage of the email behaviour.

---

Pattern 3: coverage theatre

Coverage is the percentage of code lines executed during the test run. A line can be executed without anything about it being checked.

Generated suites are very good at raising this number, because calling a function is easy and asserting the right thing is hard.

Reject:

test('renders the dashboard', () => {
  render(<Dashboard user={mockUser} />);
});

This executes hundreds of lines. It asserts nothing. It fails only if the component throws. Coverage goes up; confidence does not.

Symptoms of coverage theatre in a review:

• Tests with no expect at all
• Twelve tests that call the same function with slightly different inputs and all assert not.toThrow()
• Snapshot tests committed without anyone reading the snapshot
• Tests for getters, constructors, and constant exports
• One test per file titled "it works"

Snapshot tests deserve a specific warning. A snapshot records current output and fails when it changes. Generated snapshot tests are usually created by running the possibly-broken code and saving whatever came out. If the output was already wrong, the snapshot now protects the bug.

The counter-question for the author is not "what is your coverage?" but "which of these tests failed at least once while you were writing the feature?" A test that has never been red has never proved anything.

---

Pattern 4: tests that cannot fail

Some tests are structurally incapable of failing, usually by accident.

Async never awaited

test('rejects an invalid token', () => {
  expect(verifyToken('bad')).rejects.toThrow();  // no await, no return
});

The test function finishes before the promise settles. It passes regardless of what verifyToken does. Look for a missing await or return on every async assertion.

Assertion inside a callback that never runs

test('handles the error case', () => {
  fetchUser('missing-id').catch((err) => {
    expect(err.code).toBe('NOT_FOUND');
  });
});

If the promise resolves instead of rejecting, the callback never runs and no assertion executes. The test passes for exactly the wrong reason.

Try/catch that swallows the failure

try {
  expect(parse(input)).toEqual(expected);
} catch (e) {
  console.log('parse failed');
}

The assertion failure is caught and logged. The test reports success.

Conditional assertions

if (result.items) {
  expect(result.items.length).toBe(3);
}

When the bug removes items entirely, the assertion is skipped. The test passes on the failure it was written to catch.

---

How to actually verify a generated suite

Reading is not enough. Two cheap techniques catch what reading misses.

1. Break it on purpose. Pick the three most important tests. In the source code, invert a condition, change a + to a -, or return a hardcoded value. Run the suite. Any test that stays green is not testing that behaviour. Undo the change.
2. Run mutation testing if your language has a tool for it. Mutation testing automatically makes small changes to your code and reports which ones your tests failed to notice. It measures what coverage pretends to measure. Run it on one critical module first; it is slow.

A useful review comment, phrased so it is easy to act on:

I changed applyDiscount to always return 0 and the whole file still passed. Can we add one assertion with a hand-calculated expected total?

---

The review checklist

Run through this before approving any generated test file.

• Every test has at least one assertion
• No expected value is produced by calling the code under test
• No assertion just restates the test's own setup data
• Nothing named in the test title is mocked
• Every async assertion is awaited or returned
• No assertion sits inside an if or an unreached callback
• Exact values asserted, not toBeDefined or toBeTruthy
• At least one negative or error-path test per behaviour
• Snapshots were read by a human, not just committed
• Breaking the source on purpose turns something red

---

What to do with a bad suite

You will sometimes inherit hundreds of these. Deleting them all is politically hard and technically fine, but there is a middle path that works better.

Sort the code by risk: payments, auth, permissions, anything touching money or personal data. For those modules only, verify the tests properly using the break-it-on-purpose method, and rewrite what fails. Leave the low-risk decorative tests alone for now, but stop counting them in any coverage claim.

Then change how tests arrive. Generated tests are a fine starting draft. The rule that keeps them honest is that the author must have seen each one fail before it is merged.

---

Frequently asked questions

Should we stop generating tests altogether? No. Generation is genuinely useful for the boring parts: the setup, the table of input variations, the file skeleton. The judgement about what to assert is the part you keep for yourself.

Is high coverage always meaningless? Not meaningless, just weak. Low coverage reliably tells you something is untested. High coverage tells you lines ran. Use it to find gaps, never as proof of quality.

How do I explain this to a manager who likes the coverage number? Show one concrete example. Break a function, run the suite, show it green, and show the coverage still at 88%. That demonstration lands faster than any explanation.

What if the generated test is technically correct but tests something trivial? Delete it. Every test costs time to run and to maintain when the code changes. A test that verifies a constant is still equal to itself is a permanent small tax with no return.

Do these patterns apply to end-to-end tests too? Yes, with different shapes. The equivalents are waiting for a selector that always exists, asserting a page loaded rather than that the right data appeared, and catching errors so the run stays green. Same question applies: would this fail if the feature broke?