Playing devil's advocate with Mutation Testing
Posted: Thu Nov 29, 2007 11:17 am
As if PHPMock were not enough, I've also added a Mutation Testing engine to the mix called PHPMutagen. Just like PHPMock it's very early, but partially functional (just enough to be useful of course). Again like PHPMock, it's a generic platform capable of use in any testing/bdd framework - the default is currently PHPSpec but I'm trying to add PHPUnit or SimpleTest before requesting code review.
http://code.google.com/p/phpmutagen/
SVN: http://phpmutagen.googlecode.com/svn/trunk/
To get over the explanation bit, imagine the following code:
In standard unit testing/TDD/BDD we have the code, its test (or spec as here) and we're flying. There are however two other things we can do. The first is Code Coverage - checking what code is executed by all tests. Code not executed is likely a) not tested, or b) not executed because it's a dead execution branch (remove it!). Code Coverage is pretty common.
The second is Mutation Testing. In MT the idea is to make changes to source code (i.e. mutations) to check if the assumed broken code is detected by the test suite. If the tests do not fail - then it means that regardless of Code Coverage, we have a potential bug/code block undetectable by the test suite. So in essence, Mutation Testing tests your unit tests and specs to check if they the level of detection is sufficient.
How does it work? Taking the code above, the Mutation Testing engine would replace the original Math class with a mutated version. For example, replacing the "+" operator with "-". Presumably an add() method which now subtracts should be detected by your testing suite
. It goes further of course - you can mutate lots of different things, from operators, conditionals, PCRE patterns, variable names, string values, integer/float values, etc.
Cutting to the chase, you want to see output like:
And not:
I cheated with the second by deleting the relevant PHPSpec specification. The above is actual output from the current source code.
The theory of PHPMutagen is:
1. Identify source code
2. Run the local test suite to ensure it's clean (i.e. in a passing state)
3. Generate a list of possible mutations to source code files
4. Apply each mutation in sequence - re-run tests after each
5. Assemble mutations which did not cause the local test suite to fail.
6. Report the escaped Mutants for developer review (so can go Dalek on them
).
PHPMutagen currently has a dependency on PEAR::Text_Diff to generate unified diffs of the escaped Mutants so you see how the mutation changed the source code file. I omitted the usual /newline notice for the diff (not making patches with it are you
).
It is currently only supporting the impressive total of 2 mutations - plus operators and increment operators (+ and ++). It's pretty easy to add more - just add new PHPMutagen_Mutation subclasses and add the PHP token or string to mutate to the switch statements in PHPMutagen_MutableFile.
I'll be looking for code review later once I brush out a few final issues this evening. Anyone think this would be useful?
http://code.google.com/p/phpmutagen/
SVN: http://phpmutagen.googlecode.com/svn/trunk/
To get over the explanation bit, imagine the following code:
Code: Select all
class Math {
public function add($number1, $number2) {
return $number1 + $number2;
}
}
class DescribeMathAdding extends PHPSpec {
public function itShouldAddTwoNumbers() {
$math = new Math;
$total = $math->add(1,1);
$this->spec($total)->should->equal(2);
}
}The second is Mutation Testing. In MT the idea is to make changes to source code (i.e. mutations) to check if the assumed broken code is detected by the test suite. If the tests do not fail - then it means that regardless of Code Coverage, we have a potential bug/code block undetectable by the test suite. So in essence, Mutation Testing tests your unit tests and specs to check if they the level of detection is sufficient.
How does it work? Taking the code above, the Mutation Testing engine would replace the original Math class with a mutated version. For example, replacing the "+" operator with "-". Presumably an add() method which now subtracts should be detected by your testing suite
Cutting to the chase, you want to see output like:
Code: Select all
.......
7 Mutants born out of the mutagenic slime!
7 Mutants exterminated!
No Mutants survived! Muahahahaha!Code: Select all
....M..
7 Mutants born out of the mutagenic slime!
6 Mutants exterminated!
1 Mutant escaped; the integrity of your suite may be compromised by the following Mutants:
1)
Index: /opt/mutesting/src/PHPMock/Expectation.php
===================================================================
@@ -147,9 +147,9 @@
public function verifyCall(array $args)
{
$this->_validateOrder();
- $this->_actualCallCount++;
+ $this->_actualCallCount--;
if (!is_null($this->_exceptionToThrow)) {
if (is_array($this->_exceptionToThrow)) {
$class = $this->_exceptionToThrow[0];
$message = $this->_exceptionToThrow[1];
Happy Hunting! Remember that some Mutants may just be Ghosts (or if you want to be boring, false positives).The theory of PHPMutagen is:
1. Identify source code
2. Run the local test suite to ensure it's clean (i.e. in a passing state)
3. Generate a list of possible mutations to source code files
4. Apply each mutation in sequence - re-run tests after each
5. Assemble mutations which did not cause the local test suite to fail.
6. Report the escaped Mutants for developer review (so can go Dalek on them
PHPMutagen currently has a dependency on PEAR::Text_Diff to generate unified diffs of the escaped Mutants so you see how the mutation changed the source code file. I omitted the usual /newline notice for the diff (not making patches with it are you
It is currently only supporting the impressive total of 2 mutations - plus operators and increment operators (+ and ++). It's pretty easy to add more - just add new PHPMutagen_Mutation subclasses and add the PHP token or string to mutate to the switch statements in PHPMutagen_MutableFile.
I'll be looking for code review later once I brush out a few final issues this evening. Anyone think this would be useful?