Mutation Testing

Mutation testing asks whether your tests notice deliberate mistakes in production code. It complements coverage: executing a line is not the same as checking its behaviour.

StrykerJS changes expressions in a temporary copy of the source and runs the existing tests against each change, called a mutant. A failing test kills the mutant; passing tests leave it alive. The original source is not changed by this project’s configuration.

The goal here is better testing examples, not a perfect score. A test can catch mutations while enforcing a requirement nobody wanted. Review the intended behaviour as well as the report.

Run the Example

From the repository root, after installing dependencies:

1pnpm --filter @sveltest/website test:server --run src/lib/utils/validation.test.ts
2pnpm --filter @sveltest/website test:mutation

Open apps/website/reports/mutation/mutation.html. The adjacent mutation.json contains the machine-readable results. Both reports and the .stryker-tmp sandboxes are gitignored.

This example mutates only apps/website/src/lib/utils/validation.ts and runs its colocated validation.test.ts. It does not measure browser component tests, Svelte markup, or the whole application’s test quality.

The snippets below are independent examples for that test file, next to validation.ts, using Vite+‘s Vitest runner. The improved cases already exist in the repository; do not add duplicate copies.

Example 1: Catch a Real Surviving Mutation

The email validator’s regular expression ends with $, requiring the match to reach the end of the input. Stryker removed that anchor. The existing tests still passed: they checked malformed addresses, but none checked a valid address followed by invalid trailing text.

This test alone misses that mistake:

1import { expect, it } from 'vite-plus/test';
2import { validate_email } from './validation';
3
4it('rejects a malformed address', () => {
5	expect(validate_email('invalid-email').is_valid).toBe(false);
6});

The improvement checks a distinct behaviour, not another arbitrary email example:

1import { expect, it } from 'vite-plus/test';
2import { validate_email } from './validation';
3
4it('rejects trailing garbage after an otherwise valid address', () => {
5	expect(validate_email('[email protected]!')).toEqual({
6		is_valid: false,
7		error_message: 'Invalid email format',
8	});
9});

Without the end anchor, [email protected]! is accepted. With it, the input is rejected. The new assertion kills that specific mutant. The executable suite includes this case in its named email table, alongside the existing valid and invalid cases.

Example 2: Check the Boundary, Not Just a Short Value

Mutation testing also prompts questions that its generated mutations might not answer. Our old minimum-length test used Short1, which has six characters. That test would still pass if the minimum accidentally changed from eight to seven.

Check both sides of the boundary, while satisfying the other password rules so they cannot cause an unrelated failure:

1import { expect, it } from 'vite-plus/test';
2import { validate_password } from './validation';
3
4it.each([
5	{
6		value: 'Abcdef1',
7		is_valid: false,
8		error_message: 'Password must be at least 8 characters',
9	},
10	{
11		value: 'Abcdefg1',
12		is_valid: true,
13		error_message: '',
14	},
15])(
16	'checks the length boundary for $value',
17	({ value, is_valid, error_message }) => {
18		expect(validate_password(value)).toEqual({
19			is_valid,
20			error_message,
21		});
22	},
23);

Stryker did not generate an eight-to-seven length mutation in this run. This improvement came from reviewing the test’s claim, not from pretending the report covered every possible bug.

These are the demo’s configured password rules, not proof that a password is secure or a recommendation for an authentication policy. That is why the success test says it accepts passwords meeting the configured rules rather than calling them “strong passwords”.

Example 3: Test Debounce Timing from the Last Call

Calling a debounced function three times immediately checks that previous callbacks are cancelled. It does not establish whether the waiting period restarts after a later call.

Space the calls apart and check just before and at the final deadline:

1import {
2	afterEach,
3	beforeEach,
4	describe,
5	expect,
6	it,
7	vi,
8} from 'vite-plus/test';
9import { debounce } from './validation';
10
11describe('debounce', () => {
12	beforeEach(() => {
13		vi.useFakeTimers();
14	});
15
16	afterEach(() => {
17		vi.useRealTimers();
18	});
19
20	it('resets the waiting period after the last call', () => {
21		const callback = vi.fn();
22		const debounced = debounce(callback, 100);
23
24		debounced('first'); // t = 0
25		vi.advanceTimersByTime(50);
26		debounced('second'); // t = 50
27		vi.advanceTimersByTime(50);
28		debounced('third'); // t = 100
29
30		vi.advanceTimersByTime(99); // t = 199
31		expect(callback).not.toHaveBeenCalled();
32
33		vi.advanceTimersByTime(1); // t = 200
34		expect(callback).toHaveBeenCalledExactlyOnceWith('third');
35	});
36});

The earlier cancellation test already killed Stryker’s removal of clearTimeout. The rewrite preserves that protection and checks an additional timing requirement. A useful improvement need not change the mutation score.

See the executable validation tests for these cases and the parameterised delay checks.

Interpret the Remaining Results

The pilot caught 38 of 41 mutants before the test review and 39 of 41 afterward. Both reliable runs had no timeouts or mutation errors. These are observations from this example, not a target for other projects or a promise that future versions generate 41 mutants.

ResultWhat to investigate
KilledA test noticed the change. Check whether the assertion protects intended behaviour.
SurvivedTests passed with the change. Look for a missing case, equivalent behaviour, or an instrumentation problem.
No coverageThe mutated expression was not executed. Ask whether it represents a supported scenario or unused defensive code.
Timeout or errorInspect the diagnostic before treating it as evidence about assertion quality.

Two results remained after the review:

  • Removing ?. from result.issues[0]?.message survived. Valibot’s failed parse result guarantees a nonempty issues array, so this mutation is equivalent for results that follow that contract. Do not mock an impossible empty failure array just to kill it.
  • The fallback string 'Invalid input' was not covered. Decide whether schemas with an empty error message are a supported use case before adding a test or removing the fallback.

Decide Which Tests Earn Their Place

Use the report alongside callers, documentation, and history:

  • Keep checks that protect useful behaviour. The email compatibility cases were added deliberately during the Zod-to-Valibot migration; their unusual inputs were not a reason to delete them.
  • Rewrite tests that claim more than they check, such as the old length and debounce examples above.
  • Remove empty placeholders and unnecessary repetition. We removed 15 empty skipped tests describing speculative features, plus currency cases that mainly retested Intl.NumberFormat. Default-currency and currency-override checks remain because they verify choices made by our wrapper.

A test with no unique kills is not automatically redundant. Generated mutants represent only some possible defects. Likewise, every active test catching a mutant does not prove every expectation is justified.

How This Project Integrates Stryker

apps/website/stryker.config.json selects the production file and the Vitest runner. apps/website/vite-mutation.config.ts reuses the Vite+ configuration and selects the existing Node test file. It does not introduce a separate version of Vitest.

Two settings help with this review:

  • coverageAnalysis: 'perTest' lets Stryker select tests that cover a mutation. This is mutation coverage analysis, not the normal V8 coverage report.
  • disableBail: true records all failing tests selected for a mutant rather than crediting only the first failure. This improves the report for review but can take longer.

With Stryker 10.1.1, using its testFiles option in this pilot caused module-level schema mutations to activate too late. All 22 static mutants falsely survived. Selecting the test through the Vite+ config fixed those results. That discarded run was a tooling problem, not evidence that the tests were ineffective. Recheck the workaround when upgrading Stryker rather than treating it as a universal rule.

Consult the official Vitest runner documentation and configuration reference for the available options. This project’s narrow target and reporting choices are optional conventions, not Svelte framework defaults.

Run in CI

The mutation testing workflow runs the same command on pull requests and pushes to main, and can also be started from GitHub Actions using Run workflow. It runs only the Node example, so no Playwright browser installation or application credentials are needed. The workflow supplies fake values for the app environment variables required when SvelteKit loads the shared configuration; the utility tests do not use those values.

In a completed workflow run, download the mutation-report artifact, unzip it, and open mutation.html. mutation.json is included for programmatic inspection. Upload is attempted even if the mutation step fails, when a report may still be available.

The score is informational: thresholds.break is 0. A surviving mutant does not fail CI, but a failing baseline test, a tool failure, or a missing report does. Review the report instead of equating a green job with perfect tests. The separate unit and E2E workflows continue to run normally.