Mutation Testing for AI-Generated Tests: The CI Gate

Here is the whole demo. Four lines of source, one test.

# shop/pricing.py
def apply_discount(price, pct):
    if pct > 40:
        raise ValueError("discount too large")
    return price * (1 - pct / 100)


# tests/test_pricing.py
from shop.pricing import apply_discount


def test_smoke():
    apply_discount(100, 10)
    try:
        apply_discount(100, 50)
    except ValueError:
        pass

That test executes both branches: the first call takes the return path, the second takes the raise path, and the except ValueError swallows it so the test passes. There is nothing to fix in the coverage report, because coverage answers a different question than the one you had.

mutmut generated 10 mutants for that function and killed none of them. A mutation score of 0.0 percent at 100 percent branch coverage.

// mutants/mutmut-cicd-stats.json
{"killed": 0, "survived": 10, "total": 10, "no_tests": 0, "skipped": 0,
 "suspicious": 0, "timeout": 0, "check_was_interrupted_by_user": 0, "segfault": 0}

Then we wrote the tests twice more. The second pass added two real assertions, assert apply_discount(100, 10) == 90.0 and a pytest.raises(ValueError) around the rejected call. The third added the boundary at exactly 40, a rejection at 41, and an anchored match on both raises assertions.

Coverage is an execution proof: this line ran under some test. The mutation score is a verification proof: if this line were wrong, some test would have failed. The middle row is the one to look at, because it is what a plausible generated suite looks like: it asserts a value, it asserts an exception type, and half the ways the function can break still slip past it.

The tool rewrites one operator, literal or return value at a time, runs the tests covering that line against each version, and records whether any failed. Killed means a test caught it. Survived means every test passed while the code was wrong.

mutmut's operators are readable in its own mutation source. operator_number yields the literal plus one, so 40 becomes 41. _operator_mapping swaps relational and boolean operators in both directions: LessThan with LessThanEqual, GreaterThan with GreaterThanEqual, Equal with NotEqual, And with Or, Plus with Minus, Multiply with Divide, Power to Multiply, and the matching augmented-assignment forms. operator_assignment turns a = b into a = None. operator_string wraps a string literal in XX markers and changes its case.

PIT's default mutator group, DEFAULTS, holds eleven operators: CONDITIONALS_BOUNDARY, INCREMENTS, INVERT_NEGS, MATH, NEGATE_CONDITIONALS, VOID_METHOD_CALLS, EMPTY_RETURNS, FALSE_RETURNS, TRUE_RETURNS, NULL_RETURNS and PRIMITIVE_RETURNS. The REMOVE_CONDITIONALS family is not among them; it sits in the optional groups with CONSTRUCTOR_CALLS, NON_VOID_METHOD_CALLS, INLINE_CONSTS and REMOVE_INCREMENTS, alongside the STRONGER and ALL groups. CONDITIONALS_BOUNDARY replaces <, <=, > and >= with their boundary counterparts, the Java equivalent of the rewrite below.

After the second pass, five mutants were still alive: two that move the guard boundary, if pct > 40 to if pct >= 40 and if pct > 40 to if pct > 41, and three that attack the message, ValueError(None), "XXdiscount too largeXX" and "DISCOUNT TOO LARGE".

$ mutmut results
    shop.pricing.x_apply_discount__mutmut_1: survived
    shop.pricing.x_apply_discount__mutmut_2: survived
    shop.pricing.x_apply_discount__mutmut_3: survived
    shop.pricing.x_apply_discount__mutmut_4: survived
    shop.pricing.x_apply_discount__mutmut_5: survived

$ mutmut show shop.pricing.x_apply_discount__mutmut_1
# shop.pricing.x_apply_discount__mutmut_1: survived
--- shop/pricing.py
+++ shop/pricing.py
@@ -1,4 +1,4 @@
 def apply_discount(price, pct):
-    if pct > 40:
+    if pct >= 40:
         raise ValueError("discount too large")
     return price * (1 - pct / 100)

Read that diff as a sentence: nothing in your suite depends on whether a 40 percent discount is allowed. It is not a philosophical objection, it is a machine-generated statement about a missing assertion, and it names the line.

The statuses are not symmetrical, and the three tool families count them differently, so a mutation score is only comparable to another score from the same tool.

PIT's documentation names seven outcomes: Killed, Survived, No coverage, Non viable, Timed Out, Memory error and Run error. Its DetectionStatus enum marks detected = true for KILLED, TIMED_OUT, NON_VIABLE, MEMORY_ERROR, RUN_ERROR and EQUIVALENT, and detected = false only for SURVIVED, NO_COVERAGE, NOT_STARTED and STARTED. A mutant that blew up the JVM or hit the timeout counts toward your score exactly like a killed one, so an unstable test environment raises your mutation score.

The Stryker projects define the metric differently: Detected is killed plus timeout, Undetected is survived plus no coverage, and the mutation score is detected over valid, where valid excludes the runtime and compile errors that PIT would have counted as wins. mutmut is different again, taking total minus skipped as the denominator and killed plus timeout as the numerator.

The timeout row is why those constants matter. PIT's timeoutFactor defaults to 1.25, documented as a factor applied to the normal runtime of a test when deciding whether it is stuck in an infinite loop. StrykerJS combines timeoutMS at 5000 and timeoutFactor at 1.5 into netTimeMs * timeoutFactor + timeoutMS + overheadMs, with dryRunTimeoutMinutes at 5. mutmut computes (duration_of_original_tests + timeout_constant) * timeout_multiplier seconds, with the constant at 1.0 and the multiplier at 15.0, and labels both as unstable config that may change in any minor version.

PIT also reports two numbers worth reading side by side: mutation score is total detected over total mutations, test strength is total detected over mutations that had coverage. A large gap between them says you have untested code rather than weak assertions, and those are different repairs.

The cost of a mutation run is the mutant count multiplied by the tests that cover each mutant. It is not a complexity class, and it is smaller than the naive product because PIT and StrykerJS both do per-mutant test selection. StrykerJS coverageAnalysis defaults to perTest, which works out during the initial test run which tests cover which mutant so that only those execute per mutant; off runs every test for every mutant, and all only establishes coverage. PIT reports the same relationship in its console summary, as a tests-per-mutation ratio.

It is still a covering-test run once per mutant. That is why the pull-request version of this gate is scoped to the files the diff touched, and the same reason automated gates carry more of the load as PR volume rises: a forty-minute check on the full repository is a nightly job, not a merge condition. Each ecosystem gives you a different lever for that, and they are not interchangeable.

# setup.cfg
[mutmut]
source_paths=shop/
pytest_add_cli_args_test_selection=tests/

#!/usr/bin/env bash
set -euo pipefail

BASE="${GITHUB_BASE_REF:-main}"
FLOOR="${MUTATION_FLOOR:-60}"

git fetch --no-tags --depth=50 origin "$BASE"

patterns=()
while IFS= read -r f; do
  [ -z "$f" ] && continue
  case "$f" in */__init__.py) continue ;; esac
  mod="${f%.py}"
  patterns+=("${mod//\//.}*")
done < <(git diff --name-only "origin/$BASE...HEAD" -- 'shop/*.py' 'shop/**/*.py')

if [ ${#patterns[@]} -eq 0 ]; then
  echo "No mutatable source changed. Skipping the mutation gate."
  exit 0
fi

rm -rf mutants
mutmut run "${patterns[@]}"
mutmut export-cicd-stats
mutmut results
python ci/check_mutation_score.py "$FLOOR"

TARGETS=$(git diff --name-only "origin/${GITHUB_BASE_REF:-main}...HEAD" \
  -- 'src/main/java/**/*.java' \
  | sed -e 's|^src/main/java/||' -e 's|\.java$||' -e 's|/|.|g' \
  | paste -sd, -)

[ -z "$TARGETS" ] && { echo "No Java source changed."; exit 0; }

mvn -B test-compile org.pitest:pitest-maven:mutationCoverage \
  -DtargetClasses="$TARGETS" \
  -DmutationThreshold=70 \
  -DthresholdPrecision=2 \
  -DwithHistory=true \
  -Dthreads=4

dotnet stryker \
  --since:origin/main \
  --break-at 60 \
  --reporter "json" \
  --reporter "progress"

Adding a mutation-testing step to CI does not add a gate. PIT, mutmut, StrykerJS, Stryker.NET and Stryker4s all ship with the failure condition disabled, so the default outcome of a bad score is a green build with a report attached.

PIT gives you four gate parameters and a rounding trap. mutationThreshold, coverageThreshold and testStrengthThreshold all default to 0, maxSurviving defaults to -1, and the threshold check is skipped entirely at 0, so a PIT run with no explicit threshold can never fail one. Set one and the message is specific: Mutation score of 84 is below threshold of 85. The gates are evaluated in a fixed order, test strength, mutation score, maximum survivors, line coverage, and the first to trip is the message you get.

The trap is rounding. thresholdPrecision also defaults to 0, and PIT's percentage calculator divides with RoundingMode.HALF_UP at the requested scale. A run that killed 159 of 199 mutants is a true 79.899 percent, reports 80, and passes a threshold of 80. Set thresholdPrecision to 2 and the same run reports 79.90 and fails. Set the precision before arguing about whether the floor is 70 or 80.

<plugin>
  <groupId>org.pitest</groupId>
  <artifactId>pitest-maven</artifactId>
  <version>1.30.0</version>
  <configuration>
    <targetClasses>
      <param>com.acme.pricing.*</param>
    </targetClasses>
    <targetTests>
      <param>com.acme.pricing.*</param>
    </targetTests>
    <mutationThreshold>70</mutationThreshold>
    <thresholdPrecision>2</thresholdPrecision>
    <maxSurviving>0</maxSurviving>
    <threads>4</threads>
  </configuration>
</plugin>

maxSurviving at 0 is the strictest line in that block: it becomes unreachable as soon as one equivalent mutant lands in scope, for reasons in the limits section below.

StrykerJS fails the build on its own exit code once you set break. Thresholds default to high 80, low 60 and break null, and null is documented as preventing build failures; below break the process exits 1. Its config lives in stryker.conf.json or a sibling .js, .mjs or .cjs form, and it runs as npx stryker run.

{
  "$schema": "./node_modules/@stryker-mutator/core/schema/stryker-schema.json",
  "testRunner": "vitest",
  "coverageAnalysis": "perTest",
  "reporters": ["clear-text", "json"],
  "thresholds": { "high": 80, "low": 60, "break": 60 },
  "mutate": ["src/pricing.ts", "src/checkout.ts"]
}

The mutate array there is the diff scope, written into the file by the CI step before the run. Stryker4s follows the same shape in stryker4s.conf, thresholds high 80, low 60 and break 0, with mutate defaulting to the Scala main-source glob.

Python has no gate to configure, so you write one. mutmut 3.7.0 exits 0 whether it killed everything or nothing and prints no mutation score of its own, so both the number and the non-zero exit come from your step. Use the formula mutmut's repository uses for its score badge.

# ci/check_mutation_score.py
import json
import sys

floor = float(sys.argv[1])
with open("mutants/mutmut-cicd-stats.json") as f:
    s = json.load(f)

tested = s["total"] - s["skipped"]
score = 0.0 if tested <= 0 else (s["killed"] + s["timeout"]) / tested * 100
print(f"mutation score {score:.1f}% ({s['killed'] + s['timeout']}/{tested} killed, {s['survived']} survived)")

if score < floor:
    print(f"FAIL: mutation score {score:.1f}% is below the floor of {floor:.1f}%")
    sys.exit(1)

One more mutmut warning: the badge command documented on the project's main branch is not in the released 3.7.0, which answers Error: No such command 'badge'. Read the score out of the stats JSON instead.

A survived mutant is a better prompt than "improve the tests" because it is not a request, it is a counterexample with a line number. mutmut show prints the unified diff, and that diff is what you paste, with one sentence of instruction: write a test that fails against this diff and passes against the original.

Our second pass left five survivors, and closing them took two new tests plus an anchored match on both raises assertions. assert apply_discount(100, 40) == 60.0 kills the >= 40 mutant, which raises instead of returning 60. A rejection case at pct=41 kills the > 41 mutant, under which 41 > 41 is false and nothing is raised. match=r"^discount too large$" kills all three message mutants: None does not match, the XX-wrapped variant does not match an anchored pattern, and the upper-cased variant does not match case-sensitively.

import pytest
from shop.pricing import apply_discount


def test_applies_discount():
    assert apply_discount(100, 10) == 90.0


def test_allows_exactly_forty():
    assert apply_discount(100, 40) == 60.0


def test_rejects_forty_one():
    with pytest.raises(ValueError, match=r"^discount too large$"):
        apply_discount(100, 41)


def test_rejects_large_discount():
    with pytest.raises(ValueError, match=r"^discount too large$"):
        apply_discount(100, 50)

That anchor is load-bearing. pytest.raises matches with re.search, so a bare match="discount too large" would still let the XX-wrapped mutant survive: the pattern is found inside the wrapped string. The mutation run told us that; reading the test would not have.

Coverage across that change: unchanged, 100 percent, identical report. Mutation score: 50.0 to 100.0 percent. The loop works the same whether the tests came from a person or from the agent that wrote the code in the first place. Post the survived list as a plain CI comment with the diffs inlined; it is a mechanical artifact that sits underneath the question of where AI code reviewers fit alongside a mechanical gate rather than inside it.

Where a survivor is acceptable, mutmut takes opt-outs in the source: # pragma: no mutate for one line, # pragma: no mutate block for an indented block, and paired # pragma: no mutate start and # pragma: no mutate end for a range. That records the decision next to the code, where review can see it.

This check needs no model call and no egress. It is deterministic, runs entirely on the build agent, and does not depend on a vendor being reachable. Where an agent is allowed to draft tests but the repository sits inside a controlled perimeter, the mutation gate is what lets you accept agent-written tests without trusting the agent, which is a different control from deciding which assistants may touch controlled source at all.

Three perimeter facts follow from how these tools work.

They materialise a second full copy of your source. mutmut writes mutated source and per-file .meta results into a mutants/ directory in the working tree. StrykerJS copies the project into a sandbox directory and ships ignorePatterns to control what is copied there. PIT writes its HTML report, source included, under target/pit-reports/. In an export-controlled or residency-constrained repository those artifacts are source code and inherit the repository's handling rules: clean mutants/ in the job, and keep pit-reports and the Stryker HTML report out of any artifact store outside the perimeter.

One reporter phones home. StrykerJS ships a dashboard reporter whose default base URL is the project's hosted dashboard, with reportType full. It is not in the default reporter list of clear-text, progress and html, but a config lifted from a public example may switch it on, and in an air-gapped stage it fails the run rather than silently skipping. The safe set is clear-text plus json, parsed locally, which is what the config above uses.

Everything else works offline. mutmut treats git as a soft dependency for change detection and falls back to hashing a fixed list of build files when git is unavailable: pyproject.toml, setup.cfg, setup.py, requirements*.txt, poetry.lock, uv.lock, Pipfile and Pipfile.lock. Nothing in that path touches the network. One platform constraint: mutmut requires fork support, so a Windows build agent runs it inside WSL.

Equivalent mutants make a zero-survivor gate unreachable. PIT defines them as mutations that do not behave differently from the unmutated class, in two categories: those behaving identically, and those behaving differently but outside the scope of testing, such as logging changes. Its own enum counts EQUIVALENT as detected, with the comment that it is treated as detected although by definition it cannot be. The response to a known equivalent mutant is a pragma or an exclusion at the site, not a floor lowered across the repository.

Flaky tests read as killed. Under PIT's accounting a timeout, a memory error and a run error all count as detected, so a suite that thrashes under parallel load scores better than a stable one. If the score jumps between runs with no code change, check the timeout constants before celebrating.

Some suites produce a meaningless score. A repository whose tests are mostly integration tests over a thin domain layer shows large numbers of no-coverage mutants and a score dominated by whether the harness happened to exercise a line, which says nothing about assertions. Fix unit-level coverage of the domain layer first, then turn the gate on.

And the measurement here is one four-line function with a boundary and a string literal, exactly the shapes mutmut mutates most. Treat 0, 50 and 100 as a demonstration of the mechanism, not as a curve your repository will trace.

The floors below are starting positions to argue with in your own repository, not measured optima. What is not negotiable is the second column: the scope is the diff.

This week, take the most recent pull request in which an agent wrote the tests, run your language's mutation tool over just those source files, and read the survived list. An empty list tells you something good about that suite. Live boundary and message mutants give you the next tests to write and a number to set the floor against. Wire the failing exit code the week after, on changed files only; the rest of our AI development tools coverage has the surrounding gates.