Playing devil's advocate with Mutation Testing

Not for 'how-to' coding questions but PHP theory instead, this forum is here for those of us who wish to learn about design aspects of programming with PHP.

Moderator: General Moderators

Post Reply
User avatar
Maugrim_The_Reaper
DevNet Master
Posts: 2704
Joined: Tue Nov 02, 2004 5:43 am
Location: Ireland

Playing devil's advocate with Mutation Testing

Post by Maugrim_The_Reaper »

As if PHPMock were not enough, I've also added a Mutation Testing engine to the mix called PHPMutagen. Just like PHPMock it's very early, but partially functional (just enough to be useful of course). Again like PHPMock, it's a generic platform capable of use in any testing/bdd framework - the default is currently PHPSpec but I'm trying to add PHPUnit or SimpleTest before requesting code review.

http://code.google.com/p/phpmutagen/
SVN: http://phpmutagen.googlecode.com/svn/trunk/

To get over the explanation bit, imagine the following code:

Code: Select all

class Math {

    public function add($number1, $number2) {
        return $number1 + $number2;
    }

}

class DescribeMathAdding extends PHPSpec {

    public function itShouldAddTwoNumbers() {
        $math = new Math;
        $total = $math->add(1,1);
        $this->spec($total)->should->equal(2);
    }

}
In standard unit testing/TDD/BDD we have the code, its test (or spec as here) and we're flying. There are however two other things we can do. The first is Code Coverage - checking what code is executed by all tests. Code not executed is likely a) not tested, or b) not executed because it's a dead execution branch (remove it!). Code Coverage is pretty common.

The second is Mutation Testing. In MT the idea is to make changes to source code (i.e. mutations) to check if the assumed broken code is detected by the test suite. If the tests do not fail - then it means that regardless of Code Coverage, we have a potential bug/code block undetectable by the test suite. So in essence, Mutation Testing tests your unit tests and specs to check if they the level of detection is sufficient.

How does it work? Taking the code above, the Mutation Testing engine would replace the original Math class with a mutated version. For example, replacing the "+" operator with "-". Presumably an add() method which now subtracts should be detected by your testing suite ;). It goes further of course - you can mutate lots of different things, from operators, conditionals, PCRE patterns, variable names, string values, integer/float values, etc.

Cutting to the chase, you want to see output like:

Code: Select all

.......
7 Mutants born out of the mutagenic slime!

7 Mutants exterminated!

No Mutants survived! Muahahahaha!
And not:

Code: Select all

....M..
7 Mutants born out of the mutagenic slime!

6 Mutants exterminated!

1 Mutant escaped; the integrity of your suite may be compromised by the following Mutants:

1)
Index: /opt/mutesting/src/PHPMock/Expectation.php
===================================================================
@@ -147,9 +147,9 @@

     public function verifyCall(array $args)
     {
         $this->_validateOrder();
-        $this->_actualCallCount++;
+        $this->_actualCallCount--;
         if (!is_null($this->_exceptionToThrow)) {
             if (is_array($this->_exceptionToThrow)) {
                 $class = $this->_exceptionToThrow[0];
                 $message = $this->_exceptionToThrow[1];


Happy Hunting! Remember that some Mutants may just be Ghosts (or if you want to be boring, false positives).
I cheated with the second by deleting the relevant PHPSpec specification. The above is actual output from the current source code.

The theory of PHPMutagen is:

1. Identify source code
2. Run the local test suite to ensure it's clean (i.e. in a passing state)
3. Generate a list of possible mutations to source code files
4. Apply each mutation in sequence - re-run tests after each
5. Assemble mutations which did not cause the local test suite to fail.
6. Report the escaped Mutants for developer review (so can go Dalek on them ;)).

PHPMutagen currently has a dependency on PEAR::Text_Diff to generate unified diffs of the escaped Mutants so you see how the mutation changed the source code file. I omitted the usual /newline notice for the diff (not making patches with it are you :)).

It is currently only supporting the impressive total of 2 mutations - plus operators and increment operators (+ and ++). It's pretty easy to add more - just add new PHPMutagen_Mutation subclasses and add the PHP token or string to mutate to the switch statements in PHPMutagen_MutableFile.

I'll be looking for code review later once I brush out a few final issues this evening. Anyone think this would be useful?
User avatar
Maugrim_The_Reaper
DevNet Master
Posts: 2704
Joined: Tue Nov 02, 2004 5:43 am
Location: Ireland

Post by Maugrim_The_Reaper »

In case anyone manages to checkout the source, build it using Phing into a tarball or PEAR package (or just copy it, and amend the scripts after copying to your php dir), and is wondering how to use it:

1. It assumes you have a tests dir called "tests" or "specs"
2. It assumes you have a source dir on the same level called "src" or "lib"
3. Then you can open the command line, navigate to the parent directory of these, and "phpmutagen --workingdir /opt/mutesting" (substitute another dir for PHPMutagen to use as a working directory - default is the system temp).
4. command line options include --basedir, --workingdir, --sourcedir, --testdir. Enough rope in case your conventions are different than mine ;).
User avatar
Ambush Commander
DevNet Master
Posts: 3698
Joined: Mon Oct 25, 2004 9:29 pm
Location: New Jersey, US

Post by Ambush Commander »

This looks curiously interesting! I've heard that PHPUnit3 has this functionality too: so what's the difference?
matthijs
DevNet Master
Posts: 3360
Joined: Thu Oct 06, 2005 3:57 pm

Post by matthijs »

So now you are testing your testsuite? What's next, write testcases to test the testing of your tests?

No serious, looks interesting.
User avatar
Maugrim_The_Reaper
DevNet Master
Posts: 2704
Joined: Tue Nov 02, 2004 5:43 am
Location: Ireland

Post by Maugrim_The_Reaper »

PHPUnit3 does have Mutation Testing in SVN (it was a GSOC project) but it's not part of any release yet. The reason for PHPMutagen is to (as with PHPMock) create a standalone Mutation engine. I use SimpleTest, PHPSpec, PHPT every week and having something unbundled makes it far more useful. Secondly, PHPUnit MT relies on two PECL extensions - runkit and parse_tree. It works by mutating classes in the same process at runtime by parsing and modifying the XML output of parse_tree, reconstructing the class/method, and injecting it into the ZE using runkit. This is the common method actually outside PHP - Heckle for Ruby does the same thing almost. However, I don't like using runkit, and parse_tree has a few problems, and PHP has this cool built in Tokenizer - so I simplified the requirement to needing nothing ;), added the need for a working directory (creates a copy so mutations don't touch the original source) and ran with it. As a plus, the Tokenizer is very efficient and very easily to work with - the code is extremely simple.
So now you are testing your testsuite? What's next, write testcases to test the testing of your tests?
I hope not! :) I've really always had this opinion that Code Coverage alone was insufficient as a measure of testing success. All it really tells you is how many lines of code a testing suite executes. It doesn't say whether the tests will catch all problems. Adding Mutation Testing offers that particular confidence by ensuring your testing suite picks up on deliberately introduced errors. If it doesn't - then you found either a potential bug or a branch of execution that is untested or an assumption you rely being proven false, etc. Either way it's a valuable analysis.

The tool itself is pretty simple minded; I wrote the current version fairly quickly. It needs some refactoring, reporting classes, and adapters to other testing tools other than PHPSpec but mostly these are simple command line strings to shell_exec.
User avatar
Ambush Commander
DevNet Master
Posts: 3698
Joined: Mon Oct 25, 2004 9:29 pm
Location: New Jersey, US

Post by Ambush Commander »

That's cool!

I'm looking forward to a SimpleTest integration so I can play around with it in HTML Purifier. I certainly have encountered cases where there was a bug but all the unit tests were staying green.
User avatar
Jenk
DevNet Master
Posts: 3587
Joined: Mon Sep 19, 2005 6:24 am
Location: London

Post by Jenk »

matthijs wrote:So now you are testing your testsuite? What's next, write testcases to test the testing of your tests?

No serious, looks interesting.
It's more like a way to programatically review your tests and code base, ensuring that a) your test cases are strict, and b) you haven't got any code/beaviour without a test. :)
Post Reply