Simpletest usage question

Discussion of testing theory and practice, including methodologies (such as TDD, BDD, DDD, Agile, XP) and software - anything to do with testing goes here. (Formerly "The Testing Side of Development")

Moderator: General Moderators

Post Reply
User avatar
The Phoenix
Forum Contributor
Posts: 294
Joined: Fri Oct 06, 2006 8:12 pm

Simpletest usage question

Post by The Phoenix »

I have a collection of (different) implementations of a function.

What I need is a way to have simple test test each, and produce something similar to the following:

Test: Steve's version
1/1 test cases complete: 19 passes, 4 fails and 0 exceptions.

Test: Bob's version
1/1 test cases complete: 9 passes, 14 fails and 0 exceptions.

And so forth. I'd *love* it if you could click on the results to get the details (like Fail: testmycode -> at [/blah/foo.php line 34], but I'd be happy with the above.

I don't see an easy way to do so, but I'm fairly certain I'm missing it.

To clarify in advance, I *DO NOT* want a grouping that does:

Test: All functions
20/20 test cases complete: 2384 passes, 349 fails, and 0 exceptions.

The idea is to highlight the pass/fail count of each implementation of a single function, to help people compare competitive implementations.

Is there a simple way to do this using simpletest?
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

Sorry, I know I'm giving an answer you didn't want here, but for good reason. If you wrote two test cases for each version, a Group Test would not only tell you the number of fails, but also the name of the failing test case, line number etc.

Granted, I'd love to see SimpleTest's reporter made easier to implement since I quite often want my test results to be displayed as a series of green bars for each test it runs. I've never tried it, but I've heard that writing your own reporter is not a fun job.
User avatar
The Phoenix
Forum Contributor
Posts: 294
Joined: Fri Oct 06, 2006 8:12 pm

Post by The Phoenix »

d11wtq wrote:Sorry, I know I'm giving an answer you didn't want here, but for good reason. If you wrote two test cases for each version, a Group Test would not only tell you the number of fails, but also the name of the failing test case, line number etc.
But I want to run the same tests (23+) against all of the different implementations. Group tests wont tell me what each function did, it will tell me what the overall group did, right?
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

The Phoenix wrote:
d11wtq wrote:Sorry, I know I'm giving an answer you didn't want here, but for good reason. If you wrote two test cases for each version, a Group Test would not only tell you the number of fails, but also the name of the failing test case, line number etc.
But I want to run the same tests (23+) against all of the different implementations. Group tests wont tell me what each function did, it will tell me what the overall group did, right?
I see. From what I remember, you can group group tests if that makes sense:

Code: Select all

$stevesGroup =& new GroupTest("Steve's tests");
$stevesGroup->addTestCase(new TestOfWhatever());

$myGroup =& new GroupTest("My tests");
$myGroup->addTestCase(new TestOfWhatever());

$allTests =& new GroupTest("All Tests");
$allTests->addTestCase($stevesGroup);
$allTests->addTestCase($myGroup);

$allTests->run($reporter);
This still doesn't solve the fundamental problem though. Now the problem is how each test knows whether to use Steve's version, or your version of a function. For that, the only way I can see around it is to use a redirection function or a factory method if we're talking about an OO implementation of something.
It would be handy if it were possible to extend the group test in some way to allow attributes for test case configuration to be set/get.

Thinking out loud, something like:

Code: Select all

$myGroup =& new GroupTest("My Tests");
$myGroup->setAttribute("function_name", "myFunction");
$myGroup->addTestCase( ... );

$stevesGroup =& new GroupTest("Steve's tests");
$stevesGroup->setAttribute("function_name", "stevesFunction");
// ...
I suspect this may be more than possible without too much effort :)
User avatar
Jenk
DevNet Master
Posts: 3587
Joined: Mon Sep 19, 2005 6:24 am
Location: London

Post by Jenk »

Just write different test cases using the different implementations. The moment you start getting all Jazzy with your test cases is the moment trouble begins brewing - trouble that is not necessary, nor should be a problem because you are testing - not developing live code.

If you are using TDD, you would then need to test case your test cases to ensure they are loading the correct implementations for the right tests.

Code: Select all

class StevesTests extends UnitTestCase {
    public function testStevesFunction () 
    {/* etc */}
}

class BobsTests extends UnitTestCase {
    public function testBobsFunction ()
    {/* etc */}
}
Unit Tests are supposed to be as simple as can be.
User avatar
The Phoenix
Forum Contributor
Posts: 294
Joined: Fri Oct 06, 2006 8:12 pm

Post by The Phoenix »

Jenk wrote:Just write different test cases using the different implementations. The moment you start getting all Jazzy with your test cases is the moment trouble begins brewing - trouble that is not necessary, nor should be a problem because you are testing - not developing live code.
You completely misunderstand the problem.

There are *already existing* implementations - solutions to a problem. ie, already live code.

I need to use testing to determine the best solution to the defined problem. The problem is defined by the tests.

I thought this is what testing should enable me to do - choose between alternative implementations and find the best one.

The tests are consistent across all of the implementations. Same tests, different code solving it.

An example: One test checks for output of "Hello world". One test checks that it doesn't have formatting. Etc.

Then multiple implementations can attempt to meet that requirement.

It is key to understand that of the implementations, none currently meet all the requirements. The testing results for each will help guide development towards a correct implementation.
Unit Tests are supposed to be as simple as can be.
They are extremely simple. 23 one line tests that apply to dozens of different solutions to a problem. I just need to apply the same tests to all the different solutions.

Is this really something simpletest doesn't do well? I thought one of the big advantages of testing was being able to refactor and drop in replacements that do the job better based on testing?
User avatar
Jenk
DevNet Master
Posts: 3587
Joined: Mon Sep 19, 2005 6:24 am
Location: London

Post by Jenk »

"dropping in" replacements consists of changing:

Code: Select all

$object = new Object();
to:

Code: Select all

$object = new SomeOtherObject();
Unit Tests != Acceptance Tests / Integration Tests, which is what you seek. SimpleTest is a Unit Testing framework. Have a look for Fitnesse implementations for Integration Tests.
lastcraft
Forum Commoner
Posts: 80
Joined: Sat Jul 12, 2003 10:31 pm
Location: London

Post by lastcraft »

Hi...

There might be a way to manage this. Create the two tests, but use the XmlReporter. Then read the remote stream (if this still works - I haven't tested it lately). Then create a script that runs a test case, then runs the other.
d11wtq wrote: Granted, I'd love to see SimpleTest's reporter made easier to implement since I quite often want my test results to be displayed as a series of green bars for each test it runs. I've never tried it, but I've heard that writing your own reporter is not a fun job.
Oh dear. I've tried to make it as easy as I can :(.

yours, Marcus
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

lastcraft wrote:
d11wtq wrote: Granted, I'd love to see SimpleTest's reporter made easier to implement since I quite often want my test results to be displayed as a series of green bars for each test it runs. I've never tried it, but I've heard that writing your own reporter is not a fun job.
Oh dear. I've tried to make it as easy as I can :(.

yours, Marcus
I've never tried writing my own, but I'll give it a shot :)

You one thing I'd love too: Assertions against argument expectations if that makes sense. I can tell a Mock what I expect to see passed to it, but I need to know exactly what it is. Maybe I just want to know that the first argument passed can assertIsA(..) or assertPattern() for example. I know it can be done with a bit of custom mock work, but how easy would it be to allow assertions to be made against arguments passed to mock methods as part of the standard simpletest api? :) Unless I've missed something?

Thinking out loud...

Code: Select all

$mock->expectOnce("something");
$obj->setThing($mock);
$obj->whatever();
$args = $mock->getArgumentsAt(0, "something"); //This would be really useful
$this->assertIsA($args[0], "Foo");
lastcraft
Forum Commoner
Posts: 80
Joined: Sat Jul 12, 2003 10:31 pm
Location: London

Post by lastcraft »

Hi...
d11wtq wrote: You one thing I'd love too: Assertions against argument expectations if that makes sense. I can tell a Mock what I expect to see passed to it, but I need to know exactly what it is. Maybe I just want to know that the first argument passed can assertIsA(..) or assertPattern() for example.

Code: Select all

$mock->expectOnce('method', array(new IsaExpectation('Foo')));
That is, each argument can be replaced by an expectation in the parameter list.

yours, Marcus
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

I didn't realise that. Thanks, that's a HUGE help when I'm writing tests now :)
Post Reply