Testing test methods

Discussion of testing theory and practice, including methodologies (such as TDD, BDD, DDD, Agile, XP) and software - anything to do with testing goes here. (Formerly "The Testing Side of Development")

Moderator: General Moderators

Testing test methods

Postby josh » Mon Sep 27, 2010 11:45 pm

Who writes test for their custom assertion methods and such, and can you share that code & tests and walk us through how you did it? Did you apply TDD to writing the test utility methods themselves?
josh
DevNet Master
 
Posts: 4872
Joined: Wed Feb 11, 2004 4:23 pm
Location: Palm beach, Florida

Re: Testing test methods

Postby Jenk » Tue Oct 26, 2010 8:04 am

How complex are the custom assertions? Do they need to be custom assertions, or just a helper method on the test class (and thus can be born from abstraction rather than developed outright)

Have you got an example custom assertion you want to create?
User avatar
Jenk
DevNet Master
 
Posts: 3587
Joined: Mon Sep 19, 2005 6:24 am
Location: London

Re: Testing test methods

Postby josh » Tue Oct 26, 2010 12:10 pm

Typically I create helpers that return true or false, out of abstractions, and then pass their return value to a built in assertion for verifying.
josh
DevNet Master
 
Posts: 4872
Joined: Wed Feb 11, 2004 4:23 pm
Location: Palm beach, Florida

Re: Testing test methods

Postby Jenk » Wed Oct 27, 2010 4:30 am

Why not just use assertions in the helper instead of a return value?

Syntax: [ Download ] [ Hide ]
public function assertAreEqualUsers($expected, $returned) {
  $this->assertIsA("User", $returned);
  $this->assertEqual($expected->name, $returned->name);
  // etc.
}
User avatar
Jenk
DevNet Master
 
Posts: 3587
Joined: Mon Sep 19, 2005 6:24 am
Location: London

Re: Testing test methods

Postby josh » Wed Oct 27, 2010 8:20 pm

Because then a test failure reports that helper method as the "failure" which is confusing. There is a way to build custom assertion methods in PHPUnit without delegating, but I heard the 'proper' way to do this is to develop your assertions using unit testing, which is what I'm asking for advice on :-)

It makes sense sort of. Assertions are often more complex than the test methods. Often changing one assertion could have the side effect of silently breaking a bunch of tests (so that they always pass or something). I guess really what I'm after is some reassurance that I'm on the right track, and possibly example code :D
josh
DevNet Master
 
Posts: 4872
Joined: Wed Feb 11, 2004 4:23 pm
Location: Palm beach, Florida

Re: Testing test methods

Postby Jenk » Thu Oct 28, 2010 7:44 am

That makes perfect sense. :)

I think where I differ is that I don't see it as a need for custom assertions, I only see it as a need to refactor (i.e. extract a method). :)
User avatar
Jenk
DevNet Master
 
Posts: 3587
Joined: Mon Sep 19, 2005 6:24 am
Location: London

Re: Testing test methods

Postby McGruff » Fri Oct 29, 2010 1:39 am

josh wrote:Who writes test for their custom assertion methods and such, and can you share that code & tests and walk us through how you did it? Did you apply TDD to writing the test utility methods themselves?


I think this is very important for basic assertions which you'll use again and again. If you can't trust your test framework debugging becomes much, much harder. If you're really stuck on something you'll find yourself wondering about the test code itself.

Even use-once assertions ought to be tested but it's interesting to note that you've already got some protection without doing anything. The code you're writing tests the new (untested) test just as much as the new test tests the code. Obviously that's not perfect but to get a false positive, they both have to fail in step. That can certainly happen but it's not usually a thing which can happen easily.

Obviously there are different levels of acceptable risk. Mistakes in a forum post count aren't so bad but mistakes in a financial app can be catastrophic for a business.

I think use your judgment. If the code is simple it might be OK but if in doubt, test it.
McGruff
DevNet Master
 
Posts: 2893
Joined: Thu Jan 30, 2003 9:26 pm
Location: Glasgow, Scotland

Re: Testing test methods

Postby josh » Fri Oct 29, 2010 9:15 am

So should assertions be developed with test first or test last? Does it matter? Is there pro(s)/con(s)? Same pros/cons as testing production code I would assume? Is the general consensus not to bother writing these "meta-tests" except in exceptional situations?

The only times I ever get false positives or negatives seems to be when I forgot to make the test fail before passing (Ex. I write the test, it passes on first run, and I move on not realizing I put assertTrue() instead of assertFalse()). I guess the only thing you can do there is try not to be so absent minded, eh? These are usually really easy to fix, but happen more often than an assertion method having a bug in it's implementation.
josh
DevNet Master
 
Posts: 4872
Joined: Wed Feb 11, 2004 4:23 pm
Location: Palm beach, Florida

Re: Testing test methods

Postby josh » Fri Oct 29, 2010 10:04 am

So for example do you guys prefer:

Test Helper:
Syntax: [ Download ] [ Hide ]
$this->assertTrue( $this->levelExists('year', $year->getId()), 'when unlink year should delete year');
 


Or

Assertion Method:
Syntax: [ Download ] [ Hide ]
$this->assertLevelExists('year', $year->getId(), 'when unlink year should delete year');
 


And if so, do you use the facilities of your test framework to build a true assertion, with proper line # reporting, and robust failure message generation? Or simply delegate to a couple of lower level built in assertions?
josh
DevNet Master
 
Posts: 4872
Joined: Wed Feb 11, 2004 4:23 pm
Location: Palm beach, Florida

Re: Testing test methods

Postby VladSun » Fri Oct 29, 2010 3:20 pm

I remember I've read somewhere that:
Syntax: [ Download ] [ Hide ]
$this->assertNull($var);

should be used instead of
Syntax: [ Download ] [ Hide ]
$this->assertTrue($var == null);


So, following this logic, I think that the assertion method should be better than the test helper in your examples. I'm new to TDD though ;)
User avatar
VladSun
DevNet Master
 
Posts: 4294
Joined: Wed Jun 27, 2007 9:44 am
Location: Sofia, Bulgaria

Re: Testing test methods

Postby josh » Fri Oct 29, 2010 8:01 pm

Well if you have the assertion already developed (assertNull) its a given to use that one. However if there's a cost to creating an assertion method that doesn't yet exist. There's varying tradeoffs between test helpers & test assertion methods. Test helper methods can be conjoined into a criteria, or used from within more than one assertion method to scratch the surface

I agree the custom assertion method is always going to win in terms of readability. But what about cost is what I'm asking I guess?

I think the only 'hard & fast' rule for testing is your tests should read like documentation. So if the readability impact is negligible, what other criteria do you use to decide whether an assertion is warranted over a helper?
josh
DevNet Master
 
Posts: 4872
Joined: Wed Feb 11, 2004 4:23 pm
Location: Palm beach, Florida


Return to Testing

Who is online

Users browsing this forum: No registered users and 3 guests