Why can code coverage be a misleading metric, and what is mutation testing?

Short answer

Code coverage measures only which lines executed, not the quality or correctness of assertions. Mutation testing injects deliberate errors into the code and checks whether tests detect and "kill" them.

Explanation

You can easily reach 100% coverage with a test that calls every function and has no assert statements. Mutation testing tools such as Stryker and Pitest secretly change source symbols (for example, changing > to <, removing a function call, or replacing true with false). If tests still pass after a malicious change, the mutant survived, revealing weak and ineffective tests.

Code example

def can_vote(age: int) -> bool:
    return age >= 18

def test_can_vote():                     # 100% line coverage, yet weak
    assert can_vote(30)
    assert not can_vote(5)

# A mutation tool rewrites `>=` to `>` and re-runs the tests: they still pass → the mutant survives.
def test_can_vote_boundary():            # this assertion kills the mutant
    assert can_vote(18)

Common mistakes

• Requiring 100% code coverage in CI as a management target, encouraging developers to write superficial tests without meaningful checks just to satisfy the metric.

Follow-up questions

• What is property-based testing, and how does it find unknown edge cases compared with example-based testing?

Sources

• Mutation testing - Wikipedia