Testing test methods

[josh]

Who writes test for their custom assertion methods and such, and can you share that code & tests and walk us through how you did it? Did you apply TDD to writing the test utility methods themselves?

[Jenk]

How complex are the custom assertions? Do they need to be custom assertions, or just a helper method on the test class (and thus can be born from abstraction rather than developed outright)

Have you got an example custom assertion you want to create?

[josh]

Typically I create helpers that return true or false, out of abstractions, and then pass their return value to a built in assertion for verifying.

[Jenk]

Why not just use assertions in the helper instead of a return value?

Syntax: [ Download ] [ Hide ]

Syntax: [ Download ] [ Show ]

public function assertAreEqualUsers($expected, $returned) {
  $this->assertIsA("User", $returned);
  $this->assertEqual($expected->name, $returned->name);
  // etc.
}

[josh]

Because then a test failure reports that helper method as the "failure" which is confusing. There is a way to build custom assertion methods in PHPUnit without delegating, but I heard the 'proper' way to do this is to develop your assertions using unit testing, which is what I'm asking for advice on

It makes sense sort of. Assertions are often more complex than the test methods. Often changing one assertion could have the side effect of silently breaking a bunch of tests (so that they always pass or something). I guess really what I'm after is some reassurance that I'm on the right track, and possibly example code

[Jenk]

That makes perfect sense.

I think where I differ is that I don't see it as a need for custom assertions, I only see it as a need to refactor (i.e. extract a method).

[McGruff]

Who writes test for their custom assertion methods and such, and can you share that code & tests and walk us through how you did it? Did you apply TDD to writing the test utility methods themselves?

I think this is very important for basic assertions which you'll use again and again. If you can't trust your test framework debugging becomes much, much harder. If you're really stuck on something you'll find yourself wondering about the test code itself.

Even use-once assertions ought to be tested but it's interesting to note that you've already got some protection without doing anything. The code you're writing tests the new (untested) test just as much as the new test tests the code. Obviously that's not perfect but to get a false positive, they both have to fail in step. That can certainly happen but it's not usually a thing which can happen easily.

Obviously there are different levels of acceptable risk. Mistakes in a forum post count aren't so bad but mistakes in a financial app can be catastrophic for a business.

I think use your judgment. If the code is simple it might be OK but if in doubt, test it.

[josh]

So should assertions be developed with test first or test last? Does it matter? Is there pro(s)/con(s)? Same pros/cons as testing production code I would assume? Is the general consensus not to bother writing these "meta-tests" except in exceptional situations?

The only times I ever get false positives or negatives seems to be when I forgot to make the test fail before passing (Ex. I write the test, it passes on first run, and I move on not realizing I put assertTrue() instead of assertFalse()). I guess the only thing you can do there is try not to be so absent minded, eh? These are usually really easy to fix, but happen more often than an assertion method having a bug in it's implementation.

[josh]

So for example do you guys prefer:

Test Helper:

Syntax: [ Download ] [ Hide ]

Syntax: [ Download ] [ Show ]

$this->assertTrue( $this->levelExists('year', $year->getId()), 'when unlink year should delete year');
 

Or

Assertion Method:

Syntax: [ Download ] [ Hide ]

Syntax: [ Download ] [ Show ]

$this->assertLevelExists('year', $year->getId(), 'when unlink year should delete year');
 

And if so, do you use the facilities of your test framework to build a true assertion, with proper line # reporting, and robust failure message generation? Or simply delegate to a couple of lower level built in assertions?

[VladSun]

I remember I've read somewhere that:

Syntax: [ Download ] [ Hide ]

Syntax: [ Download ] [ Show ]

$this->assertNull($var);

should be used instead of

Syntax: [ Download ] [ Hide ]

Syntax: [ Download ] [ Show ]

$this->assertTrue($var == null);

So, following this logic, I think that the assertion method should be better than the test helper in your examples. I'm new to TDD though

[josh]

Well if you have the assertion already developed (assertNull) its a given to use that one. However if there's a cost to creating an assertion method that doesn't yet exist. There's varying tradeoffs between test helpers & test assertion methods. Test helper methods can be conjoined into a criteria, or used from within more than one assertion method to scratch the surface

I agree the custom assertion method is always going to win in terms of readability. But what about cost is what I'm asking I guess?

I think the only 'hard & fast' rule for testing is your tests should read like documentation. So if the readability impact is negligible, what other criteria do you use to decide whether an assertion is warranted over a helper?