Simpletest usage question
Moderator: General Moderators
- The Phoenix
- Forum Contributor
- Posts: 294
- Joined: Fri Oct 06, 2006 8:12 pm
Simpletest usage question
I have a collection of (different) implementations of a function.
What I need is a way to have simple test test each, and produce something similar to the following:
Test: Steve's version
1/1 test cases complete: 19 passes, 4 fails and 0 exceptions.
Test: Bob's version
1/1 test cases complete: 9 passes, 14 fails and 0 exceptions.
And so forth. I'd *love* it if you could click on the results to get the details (like Fail: testmycode -> at [/blah/foo.php line 34], but I'd be happy with the above.
I don't see an easy way to do so, but I'm fairly certain I'm missing it.
To clarify in advance, I *DO NOT* want a grouping that does:
Test: All functions
20/20 test cases complete: 2384 passes, 349 fails, and 0 exceptions.
The idea is to highlight the pass/fail count of each implementation of a single function, to help people compare competitive implementations.
Is there a simple way to do this using simpletest?
What I need is a way to have simple test test each, and produce something similar to the following:
Test: Steve's version
1/1 test cases complete: 19 passes, 4 fails and 0 exceptions.
Test: Bob's version
1/1 test cases complete: 9 passes, 14 fails and 0 exceptions.
And so forth. I'd *love* it if you could click on the results to get the details (like Fail: testmycode -> at [/blah/foo.php line 34], but I'd be happy with the above.
I don't see an easy way to do so, but I'm fairly certain I'm missing it.
To clarify in advance, I *DO NOT* want a grouping that does:
Test: All functions
20/20 test cases complete: 2384 passes, 349 fails, and 0 exceptions.
The idea is to highlight the pass/fail count of each implementation of a single function, to help people compare competitive implementations.
Is there a simple way to do this using simpletest?
- Chris Corbyn
- Breakbeat Nuttzer
- Posts: 13098
- Joined: Wed Mar 24, 2004 7:57 am
- Location: Melbourne, Australia
Sorry, I know I'm giving an answer you didn't want here, but for good reason. If you wrote two test cases for each version, a Group Test would not only tell you the number of fails, but also the name of the failing test case, line number etc.
Granted, I'd love to see SimpleTest's reporter made easier to implement since I quite often want my test results to be displayed as a series of green bars for each test it runs. I've never tried it, but I've heard that writing your own reporter is not a fun job.
Granted, I'd love to see SimpleTest's reporter made easier to implement since I quite often want my test results to be displayed as a series of green bars for each test it runs. I've never tried it, but I've heard that writing your own reporter is not a fun job.
- The Phoenix
- Forum Contributor
- Posts: 294
- Joined: Fri Oct 06, 2006 8:12 pm
But I want to run the same tests (23+) against all of the different implementations. Group tests wont tell me what each function did, it will tell me what the overall group did, right?d11wtq wrote:Sorry, I know I'm giving an answer you didn't want here, but for good reason. If you wrote two test cases for each version, a Group Test would not only tell you the number of fails, but also the name of the failing test case, line number etc.
- Chris Corbyn
- Breakbeat Nuttzer
- Posts: 13098
- Joined: Wed Mar 24, 2004 7:57 am
- Location: Melbourne, Australia
I see. From what I remember, you can group group tests if that makes sense:The Phoenix wrote:But I want to run the same tests (23+) against all of the different implementations. Group tests wont tell me what each function did, it will tell me what the overall group did, right?d11wtq wrote:Sorry, I know I'm giving an answer you didn't want here, but for good reason. If you wrote two test cases for each version, a Group Test would not only tell you the number of fails, but also the name of the failing test case, line number etc.
Code: Select all
$stevesGroup =& new GroupTest("Steve's tests");
$stevesGroup->addTestCase(new TestOfWhatever());
$myGroup =& new GroupTest("My tests");
$myGroup->addTestCase(new TestOfWhatever());
$allTests =& new GroupTest("All Tests");
$allTests->addTestCase($stevesGroup);
$allTests->addTestCase($myGroup);
$allTests->run($reporter);It would be handy if it were possible to extend the group test in some way to allow attributes for test case configuration to be set/get.
Thinking out loud, something like:
Code: Select all
$myGroup =& new GroupTest("My Tests");
$myGroup->setAttribute("function_name", "myFunction");
$myGroup->addTestCase( ... );
$stevesGroup =& new GroupTest("Steve's tests");
$stevesGroup->setAttribute("function_name", "stevesFunction");
// ...Just write different test cases using the different implementations. The moment you start getting all Jazzy with your test cases is the moment trouble begins brewing - trouble that is not necessary, nor should be a problem because you are testing - not developing live code.
If you are using TDD, you would then need to test case your test cases to ensure they are loading the correct implementations for the right tests.
Unit Tests are supposed to be as simple as can be.
If you are using TDD, you would then need to test case your test cases to ensure they are loading the correct implementations for the right tests.
Code: Select all
class StevesTests extends UnitTestCase {
public function testStevesFunction ()
{/* etc */}
}
class BobsTests extends UnitTestCase {
public function testBobsFunction ()
{/* etc */}
}- The Phoenix
- Forum Contributor
- Posts: 294
- Joined: Fri Oct 06, 2006 8:12 pm
You completely misunderstand the problem.Jenk wrote:Just write different test cases using the different implementations. The moment you start getting all Jazzy with your test cases is the moment trouble begins brewing - trouble that is not necessary, nor should be a problem because you are testing - not developing live code.
There are *already existing* implementations - solutions to a problem. ie, already live code.
I need to use testing to determine the best solution to the defined problem. The problem is defined by the tests.
I thought this is what testing should enable me to do - choose between alternative implementations and find the best one.
The tests are consistent across all of the implementations. Same tests, different code solving it.
An example: One test checks for output of "Hello world". One test checks that it doesn't have formatting. Etc.
Then multiple implementations can attempt to meet that requirement.
It is key to understand that of the implementations, none currently meet all the requirements. The testing results for each will help guide development towards a correct implementation.
They are extremely simple. 23 one line tests that apply to dozens of different solutions to a problem. I just need to apply the same tests to all the different solutions.Unit Tests are supposed to be as simple as can be.
Is this really something simpletest doesn't do well? I thought one of the big advantages of testing was being able to refactor and drop in replacements that do the job better based on testing?
"dropping in" replacements consists of changing:to:
Unit Tests != Acceptance Tests / Integration Tests, which is what you seek. SimpleTest is a Unit Testing framework. Have a look for Fitnesse implementations for Integration Tests.
Code: Select all
$object = new Object();Code: Select all
$object = new SomeOtherObject();Hi...
There might be a way to manage this. Create the two tests, but use the XmlReporter. Then read the remote stream (if this still works - I haven't tested it lately). Then create a script that runs a test case, then runs the other.
.
yours, Marcus
There might be a way to manage this. Create the two tests, but use the XmlReporter. Then read the remote stream (if this still works - I haven't tested it lately). Then create a script that runs a test case, then runs the other.
Oh dear. I've tried to make it as easy as I cand11wtq wrote: Granted, I'd love to see SimpleTest's reporter made easier to implement since I quite often want my test results to be displayed as a series of green bars for each test it runs. I've never tried it, but I've heard that writing your own reporter is not a fun job.
yours, Marcus
- Chris Corbyn
- Breakbeat Nuttzer
- Posts: 13098
- Joined: Wed Mar 24, 2004 7:57 am
- Location: Melbourne, Australia
I've never tried writing my own, but I'll give it a shotlastcraft wrote:Oh dear. I've tried to make it as easy as I cand11wtq wrote: Granted, I'd love to see SimpleTest's reporter made easier to implement since I quite often want my test results to be displayed as a series of green bars for each test it runs. I've never tried it, but I've heard that writing your own reporter is not a fun job..
yours, Marcus
You one thing I'd love too: Assertions against argument expectations if that makes sense. I can tell a Mock what I expect to see passed to it, but I need to know exactly what it is. Maybe I just want to know that the first argument passed can assertIsA(..) or assertPattern() for example. I know it can be done with a bit of custom mock work, but how easy would it be to allow assertions to be made against arguments passed to mock methods as part of the standard simpletest api?
Thinking out loud...
Code: Select all
$mock->expectOnce("something");
$obj->setThing($mock);
$obj->whatever();
$args = $mock->getArgumentsAt(0, "something"); //This would be really useful
$this->assertIsA($args[0], "Foo");Hi...
That is, each argument can be replaced by an expectation in the parameter list.
yours, Marcus
d11wtq wrote: You one thing I'd love too: Assertions against argument expectations if that makes sense. I can tell a Mock what I expect to see passed to it, but I need to know exactly what it is. Maybe I just want to know that the first argument passed can assertIsA(..) or assertPattern() for example.
Code: Select all
$mock->expectOnce('method', array(new IsaExpectation('Foo')));yours, Marcus
- Chris Corbyn
- Breakbeat Nuttzer
- Posts: 13098
- Joined: Wed Mar 24, 2004 7:57 am
- Location: Melbourne, Australia