On the devday conference I attended in September, there was this guy, Seb Rose, who asked an interesting question:
Is coverage a good metric?
When you think about it, it may not be the best. Assuming you have the following code to test:
The following unit test will give you a pretty good coverage:
But is that good? Obviously no. What if you could create code mutation and run the test again? Code mutation would be almost the same as before but with a subtle change, which would slightly change how the algorithm works and should be detected by unit test.
With that magic engine in place, we could define our new metric as percentage of failing tests, which are being run against all kinds of code mutations. Ideally, the best result is when only the original code works and the simplest semantical change breaks the test.
This became my idée fixe for the last month and it encouraged my to make a reasearch on a scary topic, which is .NET compiler platform (Roslyn). Roslyn is advertised as Compiler as a Service which in my mind translates to API that allows you to work with a source code on both syntax and semantic level.
My take on code mutation is the simplest possible: find declared const integers and increment them by one. Run the tests again to see if they fail.
Here’s my first test, no mutation applied:
Note it contains CreateCompilationForSyntaxTree, CreateAssemblyForCompilation and RunCalcTestOnAssembly, which are my helper methods for some Roslyn and Reflection magic to happen.
Code which in-memory compiles the source looks like that:
Code which builds in-memory assembly is as follows:
And finally, some reflection to run the method from the in-memory assembly:
All of above is exactly the same as the following short test. It just builds the infrastructure for the Roslyn mutation magic:
The key tool for accomplishing my task is an abstract class CSharpSyntaxRewriter, which implements visitor pattern and allows to visit a variety of syntax tree types. In my case, I’m interested in VisitLocalDeclarationStatement method. When I’m in declaration node, I need to perform some tests first, to make sure if that piece of code is what I’m interested in:
is this a constant declaration?
is this int declaration
is the variable initialized?
If any of the questions above is answered no, I’m just returning the node intact. If the given node is what I’m looking for, I read the const value, increment by one and build new node initialized with incremented value.
Don’t worry too much about that return statement. It just how you build const int two = 3; statement in Roslyn syntax API. There’s this awesome Roslynquoter page, which builds it for you.
Finally the failing test:
As you can see here, transforming this into commercial product should be easy. Just implement a ton of other mutations, provide some automation, reporting and easy to use API ;) But seriously, Roslyn is a great tool. I just can’t wait for .net vNext, which will have Roslyn as one of its building blocks. I also can’t wait for all the great tools that inevitably will follow, because the potential here is even hard to imagine.
Bonus for java guys
The similar concept is already implemented in java (as far as I can tell). Go ahead and visit pitest website for details.