Page 1 of 2

Multiple asserts per unit test

Posted: Fri Dec 21, 2007 7:00 pm
by alex.barylski
From this article: http://www.artima.com/weblogs/viewpost.jsp?thread=35578

Edit: http://blog.jayfields.com/2005/11/unit- ... lines.html

What are your opinions on this?

At first I thought, good idea, keeps things simple, then as I applied this technique to my own tests I started rethinking this philosophy...

As a general rule of thumb I can see it being good practice but I feel there are instances when multiple asserts per unit test makes sense - such as testing function with multiple exit points or return codes.

So I ask. Do you use a single unit test for each bit of funcitonality tested or do you use multiple asserts in a single unit test ever?

Cheers :)

Re: Multiple asserts per unit test

Posted: Fri Dec 21, 2007 7:46 pm
by Jenk
Hockey wrote:such as testing function with multiple exit points or return codes.
That should have multiple tests, not multiple asserts in one test. And return codes.. ugh. :P
So I ask. Do you use a single unit test for each bit of funcitonality tested or do you use multiple asserts in a single unit test ever?
A single test/spec for each piece of behaviour.

Example function that is being tested/specced:

Code: Select all

function myFunc ($a, $b) {
  return (bool) $a > $b;
}
The test case:

Code: Select all

class MyFuncTestCase extends TestCase {
  public function testShouldReturnTrue () {
    $this->assertTrue(myFunc(1, 0));
  }
  
  public function testShouldReturnFalse () {
    $this->assertFalse(myFunc(0, 1));
  }
}
The reason is because a) you haven't coded the behaviour under test yet. and b) you should be concentrating on focused behaviour traits. "The function should return true if $a is greater than $b." is separate behaviour from "The function should return false if $a is less than $b." Simplified is the key. If your requirements change later in the applications life, and the function is now supposed to return true if $a is less than $b, you can change that test, and still test the function to return true for $a > $b.

This is what is meant by one assertion per test. Focusing on behaviour. All of those blogs are referring to b/tdd so the code to satisfy the tests is yet to be completed in every scenario they talk of.

The only exception to this rule, in my opinion, is if you have a behaviour that will update multiple objects, I simply assert that each (mock)object has received the change. A Good example of this is an observer/observable test, where you want to assert that all objects registered as observers are notified.

Re: Multiple asserts per unit test

Posted: Fri Dec 21, 2007 8:34 pm
by alex.barylski
Jenk wrote:That should have multiple tests, not multiple asserts in one test. And return codes.. ugh.
Assume a model:

Code: Select all

function createUser($name)
{
  // 1) Is empty
  // 2) Is invalid
  // 3) Is duplicate

  return $code;
}
So your saying instead of something like:

Code: Select all

function testCreatePerson()
{
  assert($model->createUser('') == NAME_EMPTY);
  assert($model->createUser('*#&$') == NAME_INVALID);
  assert($model->createUser('AlreadyTaken') == NAME_DUPLICATE);

}
You would rather see something like:

Code: Select all

function testCreatePersonNameEmpty()
{
  assert($model->createUser('') == NAME_EMPTY);
}
function testCreatePersonNameInvalid()
{
  assert($model->createUser('*#&$') == NAME_INVALID);
}
function testCreatePersonNameDuplicate()
{
  assert($model->createUser('AlreadyTaken') == NAME_DUPLICATE);
}
That way you have one ASSERT per unit test??? Why, what is your reasoning behind it?

Personally I can see this being more difficult to maintain. What happens if you decide that the duplicate name is no longer a condition needed to be met. Instead of removing the single ASSERT inside a function, you need to track down test method. Not a whole lot more work I admit, but keeping multiple ASSERTS inside a method keeps things organzied to the context of the given method. Having multiple tests for a single method...just seems like more work to me...

Also. SimpleTest offers a optional second parameter in assertTrue which essentially accomplishes what I see as the only the benefit to using mutliple tests per unit. It notifies you of the exact problem in the unit, not just what unit caused the problem - which is why I originally thought a single assert per test was good idea.

--- Having read the rest of your post:

I'm not sure we see things eye to eye...

I see TDD and BDD as being quite different. classical testing is about testing results or the state of something, whereas BDD is more geared towards testing the behavior, which is what you really focus on in your reply. My question was more geared towards testing state than behavior...

As I understand it, assertions are not used for testing behavior, mocks are...this is why I'm slightly confused by your reply...

I can't help but feel I've strayed off the beaten path a bit...so I'll repeat myself. :P

What is the benefit to having single assert per unit test (given the example above) when the following does the same thing when you pass the second argument to SimpleTest?

Code: Select all

class MyFuncTestCase extends TestCase { 
  public function testShouldReturn() { 
    $this->assertTrue(myFunc(1, 0), 'TRUE'); 
    $this->assertTrue(myFunc(0, 1), 'FALSE'); 
  } 
}
On second thought, I'm not sure I like this example as I find it quite vague. Can we stick with my model example, please. :)

Cheers :)

Posted: Fri Dec 21, 2007 8:38 pm
by feyd
I would rather see exceptions, but that's me. :)

Posted: Fri Dec 21, 2007 8:57 pm
by alex.barylski
feyd wrote:I would rather see exceptions, but that's me. :)
In my experience...I would rather not use exceptions when error codes will suffice. I think an account not being created because of a duplicate existing already is something worth testing for and should be considered application logic. Exceptions are more for exceptional situations, like database connection errors, file permissions, etc...thing which demand administration attention...

Which approach is best I think depends a lot on system architecture and other things...but in my case...using exceptions sparingly for critical operations has served me well. :)

p.s-For those who wish to reply...please stick to the topic at hand...if you wish to battle it out over error codes and exceptions...please start another thread and I will contribute my experiences. :)

Cheers :)

Re: Multiple asserts per unit test

Posted: Fri Dec 21, 2007 10:22 pm
by Jenk
Hockey wrote:
Jenk wrote:That should have multiple tests, not multiple asserts in one test. And return codes.. ugh.
Assume a model:

[snipped to save scrolling]
I've already exaplained in my first reply. When using T/BDD you are focusing on behaviours, and those behaviours although possibly combined within the same function/method/object, are still separate behaviours.

As for TDD being different from BDD, well, they are not. T/BDD guru's all agree that if you are using TDD effectively (avoiding using the word "correctly",) you are doing BDD.

Re: Multiple asserts per unit test

Posted: Fri Dec 21, 2007 11:13 pm
by alex.barylski
Jenk wrote:
Hockey wrote:
Jenk wrote:That should have multiple tests, not multiple asserts in one test. And return codes.. ugh.
Assume a model:

[snipped to save scrolling]
I've already exaplained in my first reply. When using T/BDD you are focusing on behaviours, and those behaviours although possibly combined within the same function/method/object, are still separate behaviours.

As for TDD being different from BDD, well, they are not. T/BDD guru's all agree that if you are using TDD effectively (avoiding using the word "correctly",) you are doing BDD.
I've spent the entire day reading up on TDD and BDD and I have yet to see any explicit statement saying they are the same, infact I found an article by Fowler which to me suggest otherwise. ;)

Lets not argue this moot point any further, as it's clearly going to be a personal difference. :)

Posted: Fri Dec 21, 2007 11:41 pm
by Jenk
Then I suggest you read them again.

From Dave Astel's blog, who is the father of BDD: http://blog.daveastels.com/2005/07/05/a ... quote]It’s about figuring out what you are trying to do before you run off half-cocked to try to do it. You write a specification that nails down a small aspect of behaviour in a concise, unambiguous, and executable form. It’s that simple. Does that mean you write tests? No. It means you write specifications of what your code will have to do. It means you specify the behaviour of your code ahead of time. But not far ahead of time. In fact, just before you write the code is best because that’s when you have as much information at hand as you will up to that point. Like well done TDD, you work in tiny increments… specifying one small aspect of behaviour at a time, then implementing it.
[/quote]From http://behaviour-driven.org/:[quote]It must be stressed that BDD is a rephrasing of existing good practice, it is not a radically new departure. Its aim is to bring together existing, well-established techniques under a common banner and with a consistent and unambiguous terminology. BDD is very much focused on “Getting the words right” and this focus is intended to produce a vocabulary that is accurate, accessible, descriptive and consistent. [/quote]and in Maugrim's blog from this forums thread on BDD:
If you are doing TDD well, you are already doing BDD.
and from the horses mouth (Dave Astel) in a google tech talk:http://video.google.ca/videoplay?docid= ... 0081075324

Posted: Sun Dec 23, 2007 7:40 pm
by alex.barylski
Jenk wrote:Then I suggest you read them again.

From Dave Astel's blog, who is the father of BDD: http://blog.daveastels.com/2005/07/05/a ... quote]It’s about figuring out what you are trying to do before you run off half-cocked to try to do it. You write a specification that nails down a small aspect of behaviour in a concise, unambiguous, and executable form. It’s that simple. Does that mean you write tests? No. It means you write specifications of what your code will have to do. It means you specify the behaviour of your code ahead of time. But not far ahead of time. In fact, just before you write the code is best because that’s when you have as much information at hand as you will up to that point. Like well done TDD, you work in tiny increments… specifying one small aspect of behaviour at a time, then implementing it.
From http://behaviour-driven.org/:[quote]It must be stressed that BDD is a rephrasing of existing good practice, it is not a radically new departure. Its aim is to bring together existing, well-established techniques under a common banner and with a consistent and unambiguous terminology. BDD is very much focused on “Getting the words right” and this focus is intended to produce a vocabulary that is accurate, accessible, descriptive and consistent. [/quote]and in Maugrim's blog from this forums thread on BDD:
If you are doing TDD well, you are already doing BDD.
and from the horses mouth (Dave Astel) in a google tech talk:http://video.google.ca/videoplay?docid= ... 0081075324[/quote]

I think you simply misunderstood me when I said, they are not the same...

Yes, one is derived from the other so there are many commonalities, but they are not the same in the sense, that if they were the same, there would be no need for separate definitions and entire communities, would there? :P

The more I read up on them, the more differences I notice...so I'm not sure why you see them as the same.

Posted: Sun Dec 23, 2007 8:15 pm
by Jenk
Do you even bother to read? I am absolutely dumbfounded that you still state they are different. The creator of BDD himself says BDD = TDD, yet you still say they are different. Unbelievable.

Posted: Mon Dec 24, 2007 1:37 am
by alex.barylski
Jenk wrote:Do you even bother to read? I am absolutely dumbfounded that you still state they are different. The creator of BDD himself says BDD = TDD, yet you still say they are different. Unbelievable.
I don't like to read, that which doesn't make sense. ;)

They may promote the same best practices (thus if your doing TDD well, your already doing BDD) however...I think the similarities stop there. If they are so alike, why then, did the originators decide to introduce yet another method?

TDD done right is BDD...I can see that...but what about differences like concept of programming to a specification? Assertions, etc?

I see BDD as being the more polished of the two. Promoting a more abstract method of accomplishing the same task - thus it's improved and not the same.

When you write code in C and then do the exact same thing using C++ but you follow the same best practices, technically you may accomplish the same thing, but the approach is radically different, is it not???

Even though the founding members might suggest they are the same, it is nothing more than a sales technique to get more people to jump on the bandwagon. ;)

If he said "It's drastically different in every possible way but it's better" or "If your already doing TDD well, you are essentially doing BDD" which do you think works better in getting peoples interest?

Similar arguments were used when C++ was introduced:

Wikipedia: http://en.wikipedia.org/wiki/C%2B%2B
Dr. Bjarne Stroustrup developed C++ in 1979 at Bell Labs as an enhancement to the C programming language and named it "C with Classes".
The gist of the idea is the same...don't stray to far from what is already known, otherwise no one will adopt.

While you can perhaps accomplish the same result with SimpleTest and TDD practices, why would frameworks like PHPSpec be developed if you could already achieve *exactly* the same thing using the former? Because there are differences and BDD is the *next* step not the *same* step.

I suppose if you look at TDD and BDD from a strictly "practices" perspective...yes...well done TDD is BDD...but are they not more than just practices? Is it not about concept, philosophy and formalities as well?

Is it not the concept of programming to a specification that distinguishes BDD from TDD? As I am sure you are aware the word "test" is frowned upon in BDD community. I believe Dave himself tells an attendee (during that Google techtalk video) that exact same thing. call them specs not tests - test promote a less natural approach. This is a philosophical difference, but is it not a difference, and a reason to introduce "yet another" method or approach and not just call it TDD???

When you use a BDD framework like PHPSpec and you see technical/formal difference, such as the absence of assertions...is this not a difference as well? I realize that the framework is not tied to BDD but rather it's the other way around. The point is, a framework like PHPSpec promotes BDD not TDD, so in my opinion, there lies another difference.

So again, personally I consider TDD and BDD as differrent. For me personally, I see them as a collection of best practices, but also, concepts, philosophy and even more tangible technicalities.

I guess I'm just more pedantic about details, so sue me, but don't call me down and tell me I'm wrong when all you can do is quote the original authors sales pitch.

That is not what I call constructive feedback, but rather, one's personal opinion skewing the original question.

I seen this coming, as usual on this forum when opinions collide. I hope I made my point clear and we can stop this petty argument. The differences I see between BDD and TDD are important to me (again, I'm anal about details) but arguing whether my opinion is right and yours is wrong, for me personally, is moot. It benefits no one. The only reason I have persisted this far, is because I am human and your last reply was bold and I have my pride to protect. :P

So, I'm taking my ball and I'm going home. :lol:

Cheers :)

Posted: Mon Dec 24, 2007 4:30 am
by Jenk
TDD done right is BDD...I can see that...but what about differences like concept of programming to a specification? Assertions, etc?
Test Driven Design. You write tests, then code to satisfy the tests. All BDD has done is replace the word "Test" with "Specification" or "Behaviour", and replaced terms like "Assert" with "Should"

So this:

Code: Select all

class TestSomething extends TestCase {
  public function testSomethingDoesSomething {
    $this->assertTrue(true);
  }
}
Becomes:

Code: Select all

class SomethingSpec extends SpecContext {
  public function somethingShouldEqualTrue {
    $this->should(true)->equal(true);
  }
}
It is nothing more than a change of terminology, and the ability for the testing/spec framework to be near to human readable as possible.

Posted: Mon Dec 24, 2007 8:46 am
by Maugrim_The_Reaper
Test Driven Design. You write tests, then code to satisfy the tests. All BDD has done is replace the word "Test" with "Specification" or "Behaviour", and replaced terms like "Assert" with "Should"
That's a misrepresentation of BDD. Rewind by 18 months and it would hold water, but in the present day BDD has evolved primarily from real usage (particularly in Ruby where adoption has exploded) to encompass ATDD and DD (Acceptance Testing Driven Development and Domain Driven Design). In that sense the shortest summary is that it took TDD, updated to a more accurate intuitive terminology, pushed that terminology further, adopted ATDD and XP ideals (e.g. User Stories), bundled in a DSL (MF is publishing a book on DSL) and broadened it's scope into adding business value and communication value. Then focused everything into severely simplifed rules captured by the RSpec framework.

All of which is what perfection in TDD should already accomplish. But doesn't. Because TDD is not perfect - what practice ever is?

The point here is that TDD being applied effectively requires other supporting practices. In the absence of those practices, it loses much of its context, focus, and understanding. Which is why BDD exists. It ties everything into a singular discipline with superior accessibility and a set of simplified tools.

So is BDD, TDD in sheep's clothing? 'Fraid not. They are different. BDD required a whole new framework style in order to work - if it were as easy as changing a few words, then any Unit Testing library could do that. Which they've tried (without much success). The only concern BDDers have is migrating from TDD using Unit Test drivers.

You know years ago folk had this crazy idea. If something is good, then you should always do it to extremes. One of those was testing. So eXtreme Testing became popular (as has XP in general). Then another crazy guy popped up and said maybe we should write examples (as tests) before implementing? And decided like every lazy assed programmer out there (we love programming laziness) to reuse Unit Testing frameworks - IMO the reason why TDD has not worked well. It became TDD, much beloved, and rarely applied. Then another crazy SOB popped up and said, wait! This whole test thing is confusing...and it makes folk think we really are crazy! Let's boil up years of XP and TDD best practice and deliver a new dish everyone can actually understand after 5 minutes of reading.

You can guess the rest - TDD evolved out of extreme testing, BDD evolved out of TDD - they strive for the same goals in the end, but they are not the same thing! TDD is not Testing, and BDD is not TDD - but they all share very similar practices since they're the same species of animal. One might also say the reason BDD and TDD are linked so obviously is because TDD has also evolved - I'm not sure you could find any two people willing to agree on what TDD should be anymore.

Final quote after 18 months of evolutionary progress conceptualising BDD?
BDD is no longer just about “should instead of assert”, it’s evolving into a process. Emphasizing central concepts from extreme programming and domain-driven design, it’s moving toward focusing on customer stories and acceptance testing. It’s outside-in, starting at high levels of detail, rather than low-level like RSpec or Test::Unit.
From the horses mouth - read the slides, page 15 is of great interest:
http://blog.davidchelimsky.net/articles ... onf-slides

Posted: Mon Dec 24, 2007 9:16 am
by matthijs
Sounds very interesting. And logical. TDD evolved to something more, starting at higher levels instead of lower levels (user stories vs method tests). If I'm not misunderstanding.

Write another few paragraphs and you have another article Maugrim! Is there any other good documentation/resources for BDD? Even for TDD good tutorials and examples are very few. For me some practical examples would clarify most.

Posted: Mon Dec 24, 2007 11:18 am
by alex.barylski
matthijs wrote:Sounds very interesting. And logical. TDD evolved to something more, starting at higher levels instead of lower levels (user stories vs method tests). If I'm not misunderstanding.

Write another few paragraphs and you have another article Maugrim! Is there any other good documentation/resources for BDD? Even for TDD good tutorials and examples are very few. For me some practical examples would clarify most.
Maugrim has detailed manual/tutorials for using his PHPSpec framework...worth reading, although I found it easier to grasp once I found some start article and then re-read his manual (http://dev.phpspec.org/manual/).

General Introductions:
http://dannorth.net/introducing-bdd
http://behaviour-driven.org/

More Interesting:
http://www.lostechies.com/blogs/joe_oca ... pment.aspx
http://adams.id.au/blog/2007/10/what-is ... velopment/
http://blog.mootools.net/2007/10/20/wha ... pec-runner
http://video.google.com/videoplay?docid ... 0081075324

http://www.testearly.com/2007/07/16/usi ... velopment/

Also if your new to TDD/BDD like me you will need some introduction into mock objects:

http://www.ibm.com/developerworks/libra ... ktest.html
http://martinfowler.com/articles/mocksArentStubs.html

There are a few of those links with blog comments at the end...I found reading those and following their links helpful as well as they usually offer counter arguments to whatever it is the author dictates. Although not nessecarily correct, I find that it helps me to resolve my own questions when i'm split on a decision.

HTH

Cheers :)