Page 1 of 1
simpletest: stop running test on first failure
Posted: Thu Nov 27, 2008 2:18 pm
by dml
I'm sure this has come up before, but I'm not sure what to search for. I want my tests to stop on the first failure, because there's a good chance that subsequent failures are just a consequence of the state of affairs revealed by the first failure, and the error messages don't give me any more information. Is there some configuration setting for this?
Code: Select all
function test_blah(){
$a = null;
// I want the test to stop if this fails
$this->assertNotNull($a);
// Because I know these are going to fail
$this->assertEqual(99, $a["b"]);
}
Re: simpletest: stop running test on first failure
Posted: Thu Nov 27, 2008 2:50 pm
by arjan.top
it should stop at first failure by default
Re: simpletest: stop running test on first failure
Posted: Thu Nov 27, 2008 3:12 pm
by dml
It's unfortunately not working like that for me. Here's a minimal example that uses the latest version of SimpleTest (1.0.1). When I run it, it doesn't stop on failure. Any ideas?
Code: Select all
// unpacked version 1.0.1
require 'simpletest/unit_tester.php';
class MyTest extends UnitTestCase {
function test_blah(){
$a = null;
// I want the test to stop if this fails
$this->assertNotNull($a);
// Because I know these are going to fail
$this->assertEqual(99, $a["b"]);
}
}
$test = new MyTest();
$test->run(new TextReporter());
Code: Select all
OUTPUT:
MyTest
1) [NULL] should not be null at [/var/www/blah/t.php line 9]
in test_blah
2) Equal expectation fails because [Integer: 99] differs from [NULL] by 99 at [/var/www/blah/t.php line 12]
in test_blah
FAILURES!!!
Test cases run: 1/1, Passes: 0, Failures: 2, Exceptions: 0
Re: simpletest: stop running test on first failure
Posted: Fri Nov 28, 2008 3:24 am
by Jenk
This is the desired behaviour, and honestly is how it should be. Tests are not supposed to be procedural, they are a number of assertions, and failures tell us how much influence an item within our code will impact the rest of the application. In this case, we can see that $a being null will not only implicate the fact it is not 'not null' but that it will effect other behaviour(s).
Basically, a list of failures is not a stack trace, it is a list of failures. The difference may not be entirely clear at first, but after some time it is quite apparent and very useful.

Re: simpletest: stop running test on first failure
Posted: Fri Nov 28, 2008 2:47 pm
by dml
So if I understand correctly, the case you have in mind is that if component B depends on component A, then if I break A, this should break the tests not only for A but for B. I'm ok with that.
What I should have said was that I'd like a way to configure things so each test function stops running and records a failure on the first assertion failure. The case I have in mind is where assertions within a test go from general to specific - assert(account!=null); assert(account->hasOverdraftFacility()); assert(account->getOverdraftLimit()==1000); etc, etc. Rather than a functional dependency between components in the system, we're dealing with a logical dependency between assertions in the test, where if an upstream assertion fails, then a downstream assertion will either fail or its result will be meaningless. Printing out the downstream messages is just noise, and as I add more precise assertions to the tests, the error output will get noisier.
Now that I think about it, the best way of implementing those kinds of assertion sequences is probably to create an assertWithException() method that exits the function with an exception if the assertion is false.
Re: simpletest: stop running test on first failure
Posted: Mon Dec 01, 2008 3:39 am
by Jenk
Or better yet to implement the functionality that it should cease within your code, and not the test framework. In the real world, and not a test harness, would your code stop immediately when $a == null? Probably not, so perhaps you need to throw and exception within your code?

Re: simpletest: stop running test on first failure
Posted: Mon Dec 01, 2008 12:47 pm
by josh
The noise in test failures is coming from repeat code coverage, a good thing in my opinion because more bounds get checked. If you stubbed 100% of your collaborations you would get 0% noise, but in practice that's an unreasonable goal. The only time it should be a problem is if a regression occurs, during which you have the option of trying to understand + fix the code or simply rolling back to the last known working state. If you're doing TDD right then its theoretically better to just roll back. If adding a new test causes old tests to fail you most likely have a function with side effects you haven't accounted for, or some kind of global state ( singletons, static variables etc.. )
Re: simpletest: stop running test on first failure
Posted: Mon Dec 01, 2008 4:05 pm
by dml
Thanks for those insights.
Jenk wrote:Or better yet to implement the functionality that it should cease within your code, and not the test framework.
Yes, there are circumstances where this whole problem could be bypassed with a guard clause or two in the production code. I'll keep that in mind.
jshpro2 wrote:The noise in test failures is coming from repeat code coverage, a good thing in my opinion because more bounds get checked. If you stubbed 100% of your collaborations you would get 0% noise, but in practice that's an unreasonable goal.
Yes, repeat code coverage isn't such a bad thing - I'm all for my code getting run as often as possible in the test environment, with as varied as possible a range of interactions.
Re: simpletest: stop running test on first failure
Posted: Mon Dec 01, 2008 4:13 pm
by Eran
Yes, repeat code coverage isn't such a bad thing - I'm all for my code getting run as often as possible in the test environment, with as varied as possible a range of interactions.
Not necessarily a bad thing, but possibly a bad thing - it is a smell. Unit tests are meant to be as atomic for better localization - if one change breaks multiple tests you'll have a harder time finding the source of the problem.
Re: simpletest: stop running test on first failure
Posted: Mon Dec 01, 2008 6:11 pm
by josh
pytrin wrote:Not necessarily a bad thing, but possibly a bad thing - it is a smell. Unit tests are meant to be as atomic for better localization - if one change breaks multiple tests you'll have a harder time finding the source of the problem.
That's a good goal to work towards when writing tests, but its never gonna happen. A similar argument is that unit tests should cover 100% of the code, to do this they need to cover all bounds, to do this your tests need an infinite running time. My devils advocate argument is that these are just goals to work towards, but again never gonna happen on a large project
Re: simpletest: stop running test on first failure
Posted: Mon Dec 01, 2008 6:14 pm
by Eran
at the very least I try to avoid deliberately testing the same functionality multiple times..
Re: simpletest: stop running test on first failure
Posted: Mon Dec 01, 2008 7:39 pm
by josh
hah... some people are paranoid though like me
