Page 1 of 1

Simpletest usage question

Posted: Sun Jul 15, 2007 9:54 am
by The Phoenix
I have a collection of (different) implementations of a function.

What I need is a way to have simple test test each, and produce something similar to the following:

Test: Steve's version
1/1 test cases complete: 19 passes, 4 fails and 0 exceptions.

Test: Bob's version
1/1 test cases complete: 9 passes, 14 fails and 0 exceptions.

And so forth. I'd *love* it if you could click on the results to get the details (like Fail: testmycode -> at [/blah/foo.php line 34], but I'd be happy with the above.

I don't see an easy way to do so, but I'm fairly certain I'm missing it.

To clarify in advance, I *DO NOT* want a grouping that does:

Test: All functions
20/20 test cases complete: 2384 passes, 349 fails, and 0 exceptions.

The idea is to highlight the pass/fail count of each implementation of a single function, to help people compare competitive implementations.

Is there a simple way to do this using simpletest?

Posted: Sun Jul 15, 2007 10:12 am
by Chris Corbyn
Sorry, I know I'm giving an answer you didn't want here, but for good reason. If you wrote two test cases for each version, a Group Test would not only tell you the number of fails, but also the name of the failing test case, line number etc.

Granted, I'd love to see SimpleTest's reporter made easier to implement since I quite often want my test results to be displayed as a series of green bars for each test it runs. I've never tried it, but I've heard that writing your own reporter is not a fun job.

Posted: Sun Jul 15, 2007 10:21 am
by The Phoenix
d11wtq wrote:Sorry, I know I'm giving an answer you didn't want here, but for good reason. If you wrote two test cases for each version, a Group Test would not only tell you the number of fails, but also the name of the failing test case, line number etc.
But I want to run the same tests (23+) against all of the different implementations. Group tests wont tell me what each function did, it will tell me what the overall group did, right?

Posted: Sun Jul 15, 2007 11:28 am
by Chris Corbyn
The Phoenix wrote:
d11wtq wrote:Sorry, I know I'm giving an answer you didn't want here, but for good reason. If you wrote two test cases for each version, a Group Test would not only tell you the number of fails, but also the name of the failing test case, line number etc.
But I want to run the same tests (23+) against all of the different implementations. Group tests wont tell me what each function did, it will tell me what the overall group did, right?
I see. From what I remember, you can group group tests if that makes sense:

Code: Select all

$stevesGroup =& new GroupTest("Steve's tests");
$stevesGroup->addTestCase(new TestOfWhatever());

$myGroup =& new GroupTest("My tests");
$myGroup->addTestCase(new TestOfWhatever());

$allTests =& new GroupTest("All Tests");
$allTests->addTestCase($stevesGroup);
$allTests->addTestCase($myGroup);

$allTests->run($reporter);
This still doesn't solve the fundamental problem though. Now the problem is how each test knows whether to use Steve's version, or your version of a function. For that, the only way I can see around it is to use a redirection function or a factory method if we're talking about an OO implementation of something.
It would be handy if it were possible to extend the group test in some way to allow attributes for test case configuration to be set/get.

Thinking out loud, something like:

Code: Select all

$myGroup =& new GroupTest("My Tests");
$myGroup->setAttribute("function_name", "myFunction");
$myGroup->addTestCase( ... );

$stevesGroup =& new GroupTest("Steve's tests");
$stevesGroup->setAttribute("function_name", "stevesFunction");
// ...
I suspect this may be more than possible without too much effort :)

Posted: Mon Jul 16, 2007 3:45 am
by Jenk
Just write different test cases using the different implementations. The moment you start getting all Jazzy with your test cases is the moment trouble begins brewing - trouble that is not necessary, nor should be a problem because you are testing - not developing live code.

If you are using TDD, you would then need to test case your test cases to ensure they are loading the correct implementations for the right tests.

Code: Select all

class StevesTests extends UnitTestCase {
    public function testStevesFunction () 
    {/* etc */}
}

class BobsTests extends UnitTestCase {
    public function testBobsFunction ()
    {/* etc */}
}
Unit Tests are supposed to be as simple as can be.

Posted: Mon Jul 16, 2007 6:52 pm
by The Phoenix
Jenk wrote:Just write different test cases using the different implementations. The moment you start getting all Jazzy with your test cases is the moment trouble begins brewing - trouble that is not necessary, nor should be a problem because you are testing - not developing live code.
You completely misunderstand the problem.

There are *already existing* implementations - solutions to a problem. ie, already live code.

I need to use testing to determine the best solution to the defined problem. The problem is defined by the tests.

I thought this is what testing should enable me to do - choose between alternative implementations and find the best one.

The tests are consistent across all of the implementations. Same tests, different code solving it.

An example: One test checks for output of "Hello world". One test checks that it doesn't have formatting. Etc.

Then multiple implementations can attempt to meet that requirement.

It is key to understand that of the implementations, none currently meet all the requirements. The testing results for each will help guide development towards a correct implementation.
Unit Tests are supposed to be as simple as can be.
They are extremely simple. 23 one line tests that apply to dozens of different solutions to a problem. I just need to apply the same tests to all the different solutions.

Is this really something simpletest doesn't do well? I thought one of the big advantages of testing was being able to refactor and drop in replacements that do the job better based on testing?

Posted: Mon Jul 16, 2007 7:03 pm
by Jenk
"dropping in" replacements consists of changing:

Code: Select all

$object = new Object();
to:

Code: Select all

$object = new SomeOtherObject();
Unit Tests != Acceptance Tests / Integration Tests, which is what you seek. SimpleTest is a Unit Testing framework. Have a look for Fitnesse implementations for Integration Tests.

Posted: Thu Jul 19, 2007 6:10 pm
by lastcraft
Hi...

There might be a way to manage this. Create the two tests, but use the XmlReporter. Then read the remote stream (if this still works - I haven't tested it lately). Then create a script that runs a test case, then runs the other.
d11wtq wrote: Granted, I'd love to see SimpleTest's reporter made easier to implement since I quite often want my test results to be displayed as a series of green bars for each test it runs. I've never tried it, but I've heard that writing your own reporter is not a fun job.
Oh dear. I've tried to make it as easy as I can :(.

yours, Marcus

Posted: Fri Jul 20, 2007 4:52 am
by Chris Corbyn
lastcraft wrote:
d11wtq wrote: Granted, I'd love to see SimpleTest's reporter made easier to implement since I quite often want my test results to be displayed as a series of green bars for each test it runs. I've never tried it, but I've heard that writing your own reporter is not a fun job.
Oh dear. I've tried to make it as easy as I can :(.

yours, Marcus
I've never tried writing my own, but I'll give it a shot :)

You one thing I'd love too: Assertions against argument expectations if that makes sense. I can tell a Mock what I expect to see passed to it, but I need to know exactly what it is. Maybe I just want to know that the first argument passed can assertIsA(..) or assertPattern() for example. I know it can be done with a bit of custom mock work, but how easy would it be to allow assertions to be made against arguments passed to mock methods as part of the standard simpletest api? :) Unless I've missed something?

Thinking out loud...

Code: Select all

$mock->expectOnce("something");
$obj->setThing($mock);
$obj->whatever();
$args = $mock->getArgumentsAt(0, "something"); //This would be really useful
$this->assertIsA($args[0], "Foo");

Posted: Sun Jul 22, 2007 11:38 am
by lastcraft
Hi...
d11wtq wrote: You one thing I'd love too: Assertions against argument expectations if that makes sense. I can tell a Mock what I expect to see passed to it, but I need to know exactly what it is. Maybe I just want to know that the first argument passed can assertIsA(..) or assertPattern() for example.

Code: Select all

$mock->expectOnce('method', array(new IsaExpectation('Foo')));
That is, each argument can be replaced by an expectation in the parameter list.

yours, Marcus

Posted: Mon Jul 23, 2007 4:19 am
by Chris Corbyn
I didn't realise that. Thanks, that's a HUGE help when I'm writing tests now :)