Mutation Testing and AI Test Generation: Killing Surviving Mutants with Stryker

The 100% Code Coverage Trap

Many engineering teams celebrate when their continuous integration dashboard reports 95% or 100% code coverage. In reality, standard line and branch coverage is one of the easiest metrics to fake. Consider this function and its companion unit test:

// fee-calculator.ts
export function calculateTieredFee(amount: number, isVip: boolean): number {
  if (amount <= 0) return 0;
  let rate = 0.03;
  if (isVip || amount >= 10000) {
    rate = 0.015;
  }
  return amount * rate;
}

// fee-calculator.test.ts (100% coverage, 0 value)
test('calculates fee', () => {
  calculateTieredFee(500, false);
  calculateTieredFee(20000, true);
  calculateTieredFee(-10, false);
  // Notice: Zero assertions! Yet istanbul/c8 reports 100% line coverage.
});

Code coverage tells you which lines were executed by the CPU. It tells you nothing about whether your tests verify expected outputs or catch critical logic regressions. If an engineer changes amount >= 10000 to amount > 10000, or breaks the calculation entirely, the suite passes with flying colors. To expose these vulnerabilities, we turn to mutation testing and automated AI test generation.

How Mutation Testing Works: The Biological Analogy

Mutation testing inverts the testing paradigm. Instead of testing your code, it tests the quality of your tests. A mutation runner like Stryker JS systematically alters your source code in memory, introducing small intentional bugs called mutants:

• Equality Operator Mutator: Changes === to !==.
• Conditional Boundary Mutator: Changes amount >= 10000 to amount > 10000.
• Arithmetic Operator Mutator: Changes amount * rate to amount / rate.
• Block Statement Mutator: Empties the body of an if block.

After each mutation, Stryker executes the corresponding test suite:

[Original Code] ──► Stryker injects mutation (e.g. >= becomes >)
                         │
                         ▼
                 [Run Test Suite]
                    │        │
        Tests Fail  │        │  Tests Still Pass!
                    ▼        ▼
            MUTANT KILLED    MUTANT SURVIVED
            (High Quality)   (Test suite has a blind spot!)

When tests fail, the mutant is killed. When tests pass despite broken logic, the mutant survives. Surviving mutants pinpoint the exact blind spots in your test assertions.

Step 1: Installing and Configuring Stryker JS

Stryker supports modern Node.js and TypeScript test runners, including Vitest and Jest. Here is the configuration setup for a Vitest-based TypeScript project:

# Install Stryker core and Vitest runner
npm install -D @stryker-mutator/core @stryker-mutator/vitest-runner vitest typescript

Create the Stryker configuration file:

// stryker.config.json
{
  "$schema": "https://raw.githubusercontent.com/stryker-mutator/stryker-js/master/packages/api/schema/stryker-schema.json",
  "packageManager": "npm",
  "reporters": ["html", "clear-text", "progress", "json"],
  "testRunner": "vitest",
  "coverageAnalysis": "perTest",
  "mutate": [
    "src/**/*.ts",
    "!src/**/*.test.ts",
    "!src/**/*.d.ts"
  ],
  "thresholds": {
    "high": 80,
    "low": 60,
    "break": 70
  }
}

Step 2: Running Stryker and Inspecting Surviving Mutants

Execute Stryker from your terminal:

npx stryker run

Stryker generates a clear terminal summary and outputs a detailed JSON report:

# Mutation Testing Summary
Total Mutants: 14
Killed: 9
Survived: 5
Mutation Score: 64.28%

Survived Mutants Details:
File: src/fee-calculator.ts
Line 5: if (isVip || amount >= 10000)
  Mutator: EqualityOperator
  Mutated: if (isVip || amount > 10000)
  Status: Survived (No test asserted exact boundary 10,000)

Step 3: Closing Blind Spots with AI Test Generation

Surviving mutants provide the exact recipe for an AI test generator. Instead of asking an LLM to generate tests blindly, you provide the surviving mutation diff and instruct the model to write an explicit assertion that kills that specific mutant.

Here is a complete Node.js script that parses Stryker's JSON output and generates killer tests using the Anthropic API:

// scripts/generate-mutant-killers.ts
import * as fs from 'node:fs/promises';
import { createAnthropic } from '@ai-sdk/anthropic';
import { generateText } from 'ai';

const anthropic = createAnthropic({ apiKey: process.env.ANTHROPIC_API_KEY });

interface StrykerReport {
  files: Record<string, {
    source: string;
    mutants: Array<{
      id: string;
      mutatorName: string;
      status: string;
      replacement: string;
      location: { start: { line: number; column: number }; end: { line: number; column: number } };
    }>;
  }>;
}

async function generateKillerTests(): Promise<void> {
  const reportRaw = await fs.readFile('reports/mutation/mutation.json', 'utf-8');
  const report: StrykerReport = JSON.parse(reportRaw);

  for (const [filePath, fileData] of Object.entries(report.files)) {
    const survivingMutants = fileData.mutants.filter((m) => m.status === 'Survived');
    if (survivingMutants.length === 0) continue;

    console.log(`Processing ${survivingMutants.length} surviving mutants in ${filePath}...`);

    const prompt = `You are a Principal Test Automation Engineer.
Below is the TypeScript source code and a list of surviving mutants produced by Stryker JS.
Write a Vitest test suite with precise assertions that deliberately KILL every surviving mutant.

### Source Code (${filePath}):
\`\`\`typescript
${fileData.source}
\`\`\`

### Surviving Mutants:
${survivingMutants.map((m) => `- Line ${m.location.start.line}: ${m.mutatorName} replaced with '${m.replacement}'`).join('\n')}

Output only runnable Vitest test code inside a markdown block.`;

    const { text } = await generateText({
      model: anthropic('claude-3-5-sonnet-20241022'),
      prompt,
    });

    const outPath = filePath.replace('.ts', '.mutant-killer.test.ts');
    const cleanedCode = text.replace(/```typescript/g, '').replace(/```/g, '').trim();
    await fs.writeFile(outPath, cleanedCode, 'utf-8');
    console.log(`Wrote mutant killer tests to: ${outPath}`);
  }
}

generateKillerTests().catch(console.error);

The AI-Generated Killer Test Suite

Here is the targeted test suite produced by the generator to kill the surviving boundary mutant:

// src/fee-calculator.mutant-killer.test.ts
import { describe, it, expect } from 'vitest';
import { calculateTieredFee } from './fee-calculator.js';

describe('calculateTieredFee Mutation Killers', () => {
  it('asserts exact boundary condition at amount 10000 for regular users', () => {
    // Kills mutant: amount >= 10000 mutated to amount > 10000
    const feeAtBoundary = calculateTieredFee(10000, false);
    expect(feeAtBoundary).toBe(150); // 10000 * 0.015

    const feeBelowBoundary = calculateTieredFee(9999.99, false);
    expect(feeBelowBoundary).toBeCloseTo(299.9997, 2); // 9999.99 * 0.03
  });

  it('asserts zero fee for zero or negative amount inputs', () => {
    // Kills mutant: amount <= 0 mutated to amount < 0
    expect(calculateTieredFee(0, false)).toBe(0);
    expect(calculateTieredFee(-50, false)).toBe(0);
  });
});

When re-running npx stryker run, every mutant is killed and the mutation score reaches 100%.

Running Mutation Testing in CI Without Slowing Builds

Mutation testing runs dozens of test permutations, which can take several minutes on massive repos. To use Stryker in production CI pipelines effectively:

• Enable Incremental Mode: Add "incremental": true to stryker.config.json. Stryker will cache previous mutation outcomes and only test code modified in recent git commits.
• Scope to Pull Requests: Run mutation testing only on pull request branches targeting main, filtering mutated files via git diff: npx stryker run --mutate $(git diff --name-only origin/main | grep 'src/.*\.ts$').
• Prioritize Core Domain Modules: Focus mutation testing on billing calculators, authorization logic, and data transformers where silent bugs cost real money.

Combining Stryker's brutal mutation rigor with targeted AI test generation ensures that when your CI pipeline reports green, your software is genuinely verified against regressions.