AI-Written Tests Are Tautological. Coverage Lies.

Q: Why are AI-generated tests called tautological?

Because when you hand a model an implementation and ask it to write tests, the expected values in those tests are derived from the implementation itself — so the test asserts that the code does what the code does, which is true by construction and therefore proves nothing. Consider a function with an off-by-one error in a loop bound. A human writing a test from the requirement computes the expected result from the requirement and the test fails, revealing the bug. A model reading the implementation observes the actual output, asserts that output, and the test passes. The bug is now not merely undetected but actively protected: it has been written into the suite as the specification. The same applies to a function returning null where it should raise an exception, to an endpoint returning a 200 where it should return a 400, and to a rounding rule that drops a cent. In each case the assertion is a photograph of current behaviour rather than a statement about correct behaviour. This is different from a merely weak test. A thin human-written test at least draws its expected value from someone's idea of intent, however vague, so there remains a chance that intent and implementation diverge and the test catches it. A tautological test has closed that gap deliberately. It is also different from a characterisation test, which pins current behaviour on purpose as a safety net before refactoring legacy code — that is a legitimate technique with a stated purpose and a limited lifetime. The problem is characterisation tests being generated by default, at scale, and reported to leadership as coverage.

Q: Are AI-generated tests worse than having no tests at all?

In a specific and important sense, yes — because a suite that cannot detect bugs still imposes the full cost of maintaining tests, while also producing false confidence and actively resisting correct changes. Three costs make this concrete. First, false confidence: a team with 94% coverage and a green build believes it has a safety net, so it ships with less manual scrutiny, deploys with more confidence, and reviews changes to well-covered code less carefully. A team with no tests knows it has no net and compensates. The belief is more dangerous than the absence. Second, resistance to correction: when a tautological test has encoded a bug as expected behaviour, fixing that bug produces a failing test — and the default organisational reading of a failing test is that someone broke something. The likely outcomes are the fix being reverted or the test being "corrected" back to the buggy assertion, both of which are the suite defending the defect. Third, the tax on refactoring: because assertions are pinned to implementation details rather than behaviour, any structural improvement trips dozens of tests, the team learns that changing code means fixing tests, and improvement slows. The honest qualifier: tautological tests do provide genuine regression detection against accidental change. If someone unintentionally alters behaviour, the test notices. That is a real if narrow benefit, and it is why the recommendation is not to delete generated tests but to fix how they are produced — spec-first generation, test-first ordering, property-based invariants, and mutation score as the gate.

Q: How do I stop AI from writing tautological tests?

Change what you put in the context window, not which model you use. The single highest-leverage change is to withhold the implementation: describe what the function should do — the requirement, the specification, the acceptance criteria — and generate tests from that description alone. The model then has nothing to copy expected values from and must derive them from intent, which is exactly what you wanted. Run the resulting tests against the real implementation, and failures become informative: either the spec is wrong or the code is, and both are things you want to learn. The second change is ordering. Write the test before the implementation exists and tautology becomes structurally impossible, because there is nothing to derive from. This plays directly to AI's strengths — models are genuinely excellent at making a specified failing test pass, which is the labour-intensive half of test-driven development, while the specification stays human. The third change is form: property-based tests assert invariants such as round-trip equality, ordering, or conservation of value, and an invariant is a claim about what must always be true rather than about what this code returned. AI proposes candidate invariants well, and reviewing one is fast because it is a single sentence you can judge against intent. The fourth is division of labour: generated tests are excellent at fixtures, mocks, setup, teardown, and parameterisation — the tedious scaffolding — and poor at deciding expected values. Let the human own the assertion and the machine own everything around it. Finally, change the review question from "does this test look reasonable?" to "where did this expected value come from?", because tautological tests look entirely reasonable and only that question exposes them.

Q: What is mutation testing and why does it matter now?

Mutation testing measures whether your tests would actually notice a bug, by deliberately introducing small faults into your code — flipping a comparison operator, changing a boundary, removing a statement, negating a condition — and reporting which of those mutants your test suite kills. A killed mutant means at least one test failed, so the suite detects that class of fault. A surviving mutant means every test still passed while the code was wrong, which is a hole in your net at a specific location. The mutation score is the proportion killed. It matters now because generation broke the metric it replaces. Line coverage counts whether code executed during a test, which was always a weak proxy for whether a test would catch a fault, but it was cheap and roughly correlated with effort when humans wrote the tests. Generated tests decouple the two completely: a tautological suite executes every line and detects nothing, producing high coverage and a poor mutation score. That divergence is precisely the signal you need, and mutation testing is the only widely available metric that surfaces it automatically. Tooling exists across ecosystems — PIT for the JVM, Stryker for JavaScript, TypeScript, C# and Scala, mutmut and cosmic-ray for Python. The main objection is runtime, since the suite runs once per mutant and full runs on a large codebase take hours. The practical answer is incremental adoption: run mutation testing on changed files in pull requests and the full suite nightly, and start with the modules where correctness matters most rather than the whole repository. Adopt it as a gate and coverage can be demoted to a diagnostic that tells you which files nobody tested at all.

Q: Should we still use code coverage as a quality gate?

No — keep coverage as a diagnostic and gate on something that measures fault detection, because generation made coverage trivially satisfiable without the property it was proxying for. Coverage was always a measure of execution rather than verification: it tells you a line ran during a test, not that any test would fail if that line were wrong. When humans wrote tests, this weak proxy was defensible because getting coverage up required writing tests, which usually required thinking about behaviour. Generation severed that link. An assistant can raise coverage from 40% to 90% in an afternoon with tests that assert current output, and the number that leadership reads as a quality signal is now decoupled from quality entirely. This is Goodhart's problem arriving all at once rather than gradually. What coverage still tells you honestly: which files and branches have no tests touching them at all. That is genuinely useful — a zero-coverage module is a real gap, and no metric replaces knowing where you have nothing. Keep it visible for that purpose. What should replace it as a gate: mutation score, which measures whether the suite detects injected faults and cannot be gamed by asserting current behaviour. If mutation testing is too expensive to adopt immediately, gating on nothing is preferable to gating on coverage, because a false quality signal is worse than an absent one. Two organisational notes. Removing a coverage gate reads as lowering standards unless you explain the reasoning, so make the argument explicitly with a concrete example of a high-coverage tautological test that catches nothing. And if a regulatory or contractual regime mandates a coverage threshold, meet it while reporting mutation score alongside as the metric your team actually manages to.

Q: What are property-based tests and why can they not be tautological?

A property-based test asserts an invariant — something that must hold for every valid input — and lets the framework generate large numbers of inputs, including adversarial edge cases, to try to falsify it. Familiar examples: reversing a list twice returns the original list; serialising and then parsing a value returns something equal to the input; a sort produces a permutation of the input in non-decreasing order; a discount never makes a total negative; withdrawing then depositing the same amount leaves a balance unchanged. Frameworks such as Hypothesis for Python, fast-check for JavaScript and TypeScript, and QuickCheck-derived libraries elsewhere also shrink failures to a minimal reproducing case, which usually points straight at the bug. They resist tautology structurally because the property is a claim about what must always be true, expressed independently of how the code computes anything. You cannot derive "round-trip returns the original" by reading an implementation and copying its output; it is a statement about the relationship between two operations. And because inputs are generated rather than chosen, the test explores regions no author thought about — empty collections, Unicode, extreme numeric values, duplicate keys, values near boundaries — which is exactly where example-based tests are thinnest. AI fits here well: models are good at proposing candidate invariants for a domain, and reviewing a proposed invariant is fast because it is one sentence you can check against your intent, unlike reviewing forty generated example tests. The honest limits: not everything has clean invariants, business logic that is genuinely a table of arbitrary rules is better tested by example, and a badly chosen property can be vacuously true. Property-based tests complement example-based ones rather than replacing them — but they are the one form that cannot be written by reading the implementation.

Q: How should code review handle AI-generated tests?

Change the question. Reviewing generated tests the way you review generated code — scanning for plausibility, structure, and naming — fails completely here, because tautological tests are well-structured, plausibly named, and comprehensively organised. They look better than most hand-written tests. The only question that exposes them is: where did this expected value come from? An expected value copied from a test run or inferred from reading the implementation is the defect. An expected value derived from a requirement, a specification, a standard, or an independently computed result is fine. Practical review heuristics that follow from that. Ask whether you can justify the assertion without looking at the implementation — if not, the test is describing rather than specifying. Be suspicious of magic numbers with no stated provenance, especially oddly specific ones, since they usually came from an observed output. Treat exhaustive assertions on entire response objects as a smell, because asserting every field pins implementation details that were never contract. Check whether error paths assert the correct behaviour or merely the current one; "returns null on invalid input" is frequently a bug that got enshrined. Look for snapshot tests, which are tautology by construction. And confirm at least some tests would fail if the implementation were subtly wrong — if you cannot imagine a bug the test catches, it catches none. Two process-level supports make this sustainable, since reviewer attention does not scale with generation volume. Require pull requests to state where expected values came from, which is cheap to write and immediately revealing. And run mutation testing on changed files in CI so a suite that cannot fail is caught mechanically rather than depending on a reviewer noticing.

Q: Where are AI-generated tests genuinely useful?

Everywhere the expected value is not the interesting part — which is most of the volume of a test suite and most of the tedium. Test data and fixtures: realistic objects, valid and invalid variants, edge-case inputs, and factory functions are mechanical work that models do quickly and well. Mocks and stubs: implementing an interface with plausible canned responses is exactly the kind of structured, patterned code generation excels at, and getting it wrong is usually obvious immediately. Setup and teardown: database seeding, container lifecycle, fixture composition, and cleanup are error-prone boilerplate where a model saves real time. Parameterisation: given one carefully written example test, expanding it into a table-driven test across many input combinations is safe, because the human already decided what correct behaviour is and the expansion only varies inputs. Test refactoring: extracting helpers, removing duplication, and migrating between frameworks are mechanical transformations with verifiable results. Candidate invariants for property-based testing: models propose these well, and each is one sentence to review. Descriptions and naming: turning intent into readable test names is genuinely helpful for suite legibility. The line running through all of these is that the human supplies the specification — what correct means — and the machine supplies everything around it. The cases to avoid are the mirror image: asking a model to determine expected values by reading an implementation, generating tests to hit a coverage number, and generating snapshot tests, all of which produce assertions derived from behaviour rather than from intent. Used on the right side of that line, generated tests measurably speed up test writing without degrading what the suite detects.

Satyam Kumar

Zurück zum Blog

ai-architecture

Your AI Wrote 400 Tests and Coverage Hit 94%. None of Them Would Catch a Bug

By Satyam KumarJuly 29, 202610 min read

ai generated tests test coverage mutation testing property based testing software testing test driven development ai code generation code quality testing strategy hypothesis pitest contract testing software engineering ci cd developer productivity 2026

Your AI Wrote 400 Tests and Coverage Hit 94%. None of Them Would Catch a Bug

Frequently Asked Questions

Artikel teilen

Twitter LinkedIn WhatsApp

Satyam Kumar

Founder & AI Architect, AppScale LLP

KI & Cloud Architekt. Ich helfe Teams, Systeme zu bauen, die auf Millionen skalieren.

LinkedIn GitHub

Your AI Wrote 400 Tests and Coverage Hit 94%. None of Them Would Catch a Bug

Frequently Asked Questions

Artikel teilen

Comments

Leave a comment