Mutation Testing with Stryker

For me, testing has two main purposes:

• To prove what I’ve just added works
• To make sure it still works when someone changes something in the future

Okay, there’s definitely more benefits to testing than this - but these, for me, are fundamental.

Mutation testing helps with the second one. If changes are made to the codebase, tests should fail. If they don’t, you likely have missing tests.

I use Stryker as my tool of choice here, because I’m in .NET land. Stryker takes your production code, and starts making thousands of little tweaks one at a time - inverting booleans or comparisons, emptying collections in initialisers - loads of things. The full list of “mutations” is here.

And then, it tells you which changes didn’t cause test failures, so you know where your testing gaps might be. Handy!

Say you have the following method in your production code:

public static string GetGrade(int score)
{
    if (score >= 90) return "A";
    if (score >= 80) return "B";
    if (score >= 70) return "C";
    if (score >= 60) return "D";
    return "F";
}

Okay, not the best approach to this in my opinion, but let’s go with it. Stryker would change those >= comparisons one at a time to, say, >. If you don’t have tests that cover the boundaries here (like a score of 90, asserting a return value of A) this will highlight your mistake.

As more and more code is produced by LLMs, I believe tests that prevent regressions become more important. We all know they sometimes churn out rubbish, or make assumptions. Maybe the diff that came back was huge, and you didn’t quite pay it the attention it deserved. Having detailed tests are going to pay off in these situations. They also give the LLM tighter guardrails while it’s implementing things - broken tests help nudge them in the right direction (as long as they don’t just rewrite the test...).