Suite Tester 1.0 Release Candidate

Discussion of testing theory and practice, including methodologies (such as TDD, BDD, DDD, Agile, XP) and software - anything to do with testing goes here. (Formerly "The Testing Side of Development")

Moderator: General Moderators

User avatar
Nathaniel
Forum Contributor
Posts: 396
Joined: Wed Aug 31, 2005 5:58 pm
Location: Arkansas, USA

Suite Tester 1.0 Release Candidate

Post by Nathaniel »

For all of us SimpleTest users, I've got a Suite Tester release candidate up. I've been using the 0.4 version myself for half of a year, and it's been incredibly helpful. I didn't have much of a workload this past week so I decided to recode it from scratch.

I've put a page about Suite Tester with some demos, so if you have a spare half-hour over the weekend, I'd appreciate it if you'd give it a go and let me know what you think :) Once I have some initial community feedback I'd like to publicize it a bit more.

Thanks,
Nathaniel
User avatar
Maugrim_The_Reaper
DevNet Master
Posts: 2704
Joined: Tue Nov 02, 2004 5:43 am
Location: Ireland

Post by Maugrim_The_Reaper »

You bought me with the first line on the web page...:)
Conveniently test an entire application or a single unit in one place!
I didn't quite follow the introduction at first. Am I right in saying it tests the entire suite at the start, and then updates individual tests as you direct? That by itself is a time saver...

- If you add Group folding (let me hide those I already know are passing) it would make navigation easier. I have test suites with a lot more than 7 test cases. Scrolling down to click a test case, and then back up to see the results will get tiresome eventually ;).
- Also it would be nice for an entire Group to be capable of a rerun.
- Optional display of a test's detailed output by running it in a popup/floating-div (a separate optional link alongside the current test title links maybe). Not as important as the main summary view but it's output I often go looking for.
- Is $_GET['test'] sanitised before hitting the include() statement? The statement also has the @ error suppressor for some reason.
- May need to adapt to super-long test names without breaking the page width.
- I'd like to see more obvious highlighting of failing test cases - like the red bar effect instead of just the red text. I know you have a CSS file in the package so it's probably just a minor point.

Nice going!

It looks like a really cool tool.
User avatar
Nathaniel
Forum Contributor
Posts: 396
Joined: Wed Aug 31, 2005 5:58 pm
Location: Arkansas, USA

Post by Nathaniel »

Hey Maugrim!
Maugrim_The_Reaper wrote:You bought me with the first line on the web page...:)

:)
I didn't quite follow the introduction at first. Am I right in saying it tests the entire suite at the start, and then updates individual tests as you direct?
Yup. Of course, if you want to reload every test, just hit F5. A "Reload All" button is planned for the next version.
If you add Group folding (let me hide those I already know are passing) it would make navigation easier. I have test suites with a lot more than 7 test cases. Scrolling down to click a test case, and then back up to see the results will get tiresome eventually ;).

Ooh, that's a great idea! Would you make it fold automatically if the group passes, then? With a "toggle folding" link next to it?
Also it would be nice for an entire Group to be capable of a rerun.

Another good idea. Hopefully it'll make it into the next version.
Optional display of a test's detailed output by running it in a popup/floating-div (a separate optional link alongside the current test title links maybe). Not as important as the main summary view but it's output I often go looking for.

I'm not quite getting this one. Do you want this extra link to spit out all of the passing tests? That could be useful.
Is $_GET['test'] sanitised before hitting the include() statement? The statement also has the @ error suppressor for some reason.

Winces no... I usually have all of my testing software locked in a password-protected directory and the thought didn't occur to me.
As for the @, I think I added that because I didn't like seeing a page full of include errors when a test file was missing. I'll take it out though. Thanks for finding that.
May need to adapt to super-long test names without breaking the page width.

Hmm, I don't quite understand. Like make the test name "A Test of ... Model"? I guess I'll have to populate a suite and try to replicate the problem you had.
I'd like to see more obvious highlighting of failing test cases - like the red bar effect instead of just the red text. I know you have a CSS file in the package so it's probably just a minor point.

Ok. Easily enough done. :) Is this what you were looking for? That'll make it into the 1.0 release; no need to wait for 2.0 there.
Nice going! It looks like a really cool tool.
Thanks, and thanks for the feedback :)
User avatar
Nathaniel
Forum Contributor
Posts: 396
Joined: Wed Aug 31, 2005 5:58 pm
Location: Arkansas, USA

Post by Nathaniel »

I have updated the Suite Tester page with Release Candidate 2. The Version History details the changes.
User avatar
RobertGonzalez
Site Administrator
Posts: 14293
Joined: Tue Sep 09, 2003 6:04 pm
Location: Fremont, CA, USA

Post by RobertGonzalez »

That is pretty neat Nathaniel. I think I am going to be playing with this muchly over the next few days.
User avatar
Ambush Commander
DevNet Master
Posts: 3698
Joined: Mon Oct 25, 2004 9:29 pm
Location: New Jersey, US

Post by Ambush Commander »

It looks interesting. I personally hacked up a little test-suite selector for my own project, but if Suite Tester offers more benefits I'll use it!

Question: is there an AJAX way to refresh the entire test suite, or do I have to reload? Also, is there a way to continually refresh a test result every, say, two seconds? With that, I wouldn't have to continually switch to my browser and click refresh every time I want to test a code change! The flickering that happens when you load reload a test is slightly annoying, but not major.
User avatar
Nathaniel
Forum Contributor
Posts: 396
Joined: Wed Aug 31, 2005 5:58 pm
Location: Arkansas, USA

Post by Nathaniel »

Ambush Commander wrote:Question: is there an AJAX way to refresh the entire test suite, or do I have to reload?
Currently we have to reload. A "Refresh All" button is in the feature queue, however.
Also, is there a way to continually refresh a test result every, say, two seconds?

Interesting idea. Prototype has a "PeriodicalUpdater" class built into it, so it shouldn't be hard to implement.

What about "on window focus, refresh the last test run"? Firebug shows the test cases taking about 60ms to load, so lag time would be negligent.
The flickering that happens when you load reload a test is slightly annoying, but not major.
Do you mean when you f5 the page, or when you click an individual test case to be rerun?
User avatar
Ambush Commander
DevNet Master
Posts: 3698
Joined: Mon Oct 25, 2004 9:29 pm
Location: New Jersey, US

Post by Ambush Commander »

What about "on window focus, refresh the last test run"? Firebug shows the test cases taking about 60ms to load, so lag time would be negligent.
That sounds good, although, you probably want to give the user more control over which tests to refresh.
Do you mean when you f5 the page, or when you click an individual test case to be rerun?
When individual test-cases get rerun. The associated messages get erased, which often moves the menu widget around: the menu of test cases should stay in the same position.
sike
Forum Commoner
Posts: 84
Joined: Wed Aug 02, 2006 8:33 am

Post by sike »

looks good nathaniel (:

as most other unit testers i have whipped up my own test selector featuring a hierarchical view and a basic code coverage display. it would be nice if you could incorporate these features because then i would definitely switch (:

chris

ps. i have put a little demo on http://code.refactored.de/test/external ... /index.php
matthijs
DevNet Master
Posts: 3360
Joined: Thu Oct 06, 2005 3:57 pm

Post by matthijs »

Ok, I have a running simpletest lib, and get the suite-tester running, but I only get "failed, with no response" results. While my simpletest results are definitely green.

My current setup is:
/suite-tester/
/tests/all-tests.php
/tests/login_test.php
.. etc
/tests/simpletest/

Now I should edit the correct paths in suite.js. But whichever path I choose, it doesn't find my tests. What should I change in my setup?
User avatar
Nathaniel
Forum Contributor
Posts: 396
Joined: Wed Aug 31, 2005 5:58 pm
Location: Arkansas, USA

Post by Nathaniel »

Hi matthijs,

The "failed, with no response" error means that library/test-runner.php is returning a blank file.

The most probable cause for this is because it's not reporting errors - probably the error generated when it can't find the file you want it to test.

Would you please open library/test-runner.php, find line 14, and insert this code?

Code: Select all

error_reporting(E_ALL);
ini_set('display_errors', '1');
Then try running Suite Tester again.

Also, can you tell me what version of PHP you are using? I haven't tested it on PHP 5, now that I think about it.

Hey Chris, glad you like it. I need to go at the moment, but I'll reply to you in a bit. Thanks for the comment!

- Nathaniel[/syntax]
matthijs
DevNet Master
Posts: 3360
Joined: Thu Oct 06, 2005 3:57 pm

Post by matthijs »

Ok, I added the error reporting, and, after changing the path again, get a succesful running of the examples, but an error for my own tests:
Myclimbing Test Case failed. Response Text:
Fatal error: Cannot redeclare class fieldexpectation in /Applications/MAMP/htdocs/tests/simpletest/web_tester.php on line 23
3/3 test cases complete: 2 passes, 0 fails, and 1 exceptions.
I'm running php 5.1
User avatar
Nathaniel
Forum Contributor
Posts: 396
Joined: Wed Aug 31, 2005 5:58 pm
Location: Arkansas, USA

Post by Nathaniel »

Ok, so your test files are include()ing the simpletest files. If you look down in test-runner.php, I require all the files you should need for you.

Two options:

1. remove these lines from test-runner.php:

Code: Select all

require(SIMPLETEST_DIR . 'unit_tester.php');
require(SIMPLETEST_DIR . 'web_tester.php');
require(SIMPLETEST_DIR . 'reporter.php');
require(SIMPLETEST_DIR . 'mock_objects.php');
2. Remove the require()/include() calls from your unit tests.

Cheers,
- Nathaniel
matthijs
DevNet Master
Posts: 3360
Joined: Thu Oct 06, 2005 3:57 pm

Post by matthijs »

Ok, thanks.

Option 1:
Example Test Case failed. Response Text:
Fatal error: Class 'TestSuite' not found in /Applications/MAMP/htdocs/suite-tester/library/test-runner.class.php on line 8
Myclimbing Test Case failed. Response Text:
Fatal error: Class 'TestSuite' not found in /Applications/MAMP/htdocs/suite-tester/library/test-runner.class.php on line 8
A Test Inside a Group failed. Response Text:
Fatal error: Class 'TestSuite' not found in /Applications/MAMP/htdocs/suite-tester/library/test-runner.class.php on line 8
3/3 test cases complete: 0 passes, 0 fails, and 3 exceptions
Option 2:
Myclimbing Test Case failed. Response Text:
Warning: TestSuite::include(web_test.php) [function.TestSuite-include]: failed to open stream: No such file or directory in /Applications/MAMP/htdocs/tests/simpletest/test_case.php on line 461

Warning: TestSuite::include() [function.include]: Failed opening 'web_test.php' for inclusion (include_path='.:/Applications/MAMP/bin/php5/lib/php') in /Applications/MAMP/htdocs/tests/simpletest/test_case.php on line 461
All tests
Fail: web_test.php -> Bad TestSuite [web_test.php] with error [Failed opening 'web_test.php' for inclusion (include_path='.:/Applications/MAMP/bin/php5/lib/php')]
0/0 test cases complete: 0 passes, 1 fails and 0 exceptions.

Warning: Cannot modify header information - headers already sent by (output started at /Applications/MAMP/htdocs/tests/simpletest/test_case.php:461) in /Applications/MAMP/htdocs/suite-tester/library/json-reporter.php on line 40
{ "messages" : [ { "type" : "exception", "msg" : "../../tests/all_tests.php -> Bad TestSuite: ../../tests/all_tests.php has no runnable test cases." } ], "passes" : "0" }
3/3 test cases complete: 2 passes, 0 fails, and 1 exceptions.
Maybe we could solve this if you'd tell me how the setup should look like. As my simpletest is setup now:
/htdocs/
/htdocs/suite-tester/
/htdocs/tests/
/htdocs/tests/all_tests.php
/htdocs/tests/log_tests.php
/htdocs/tests/login_tests.php
...
/htdocs/tests/simpletest/

the tests works fine if I run them directly by calling /tests/all_tests.php

(I have been working with PHP for a while now, but somehow I have a love-Hate relationship with includes and requires. It never goes well...)
User avatar
Nathaniel
Forum Contributor
Posts: 396
Joined: Wed Aug 31, 2005 5:58 pm
Location: Arkansas, USA

Post by Nathaniel »

The way you have the hierarchy set up shouldn't matter so long as everything points to each other correctly.

Could you pm me a link to your site so I can take a peek at stuff?

If not: It sounds like you're running the tests in all_tests.php (etc). all_tests.php, log_tests.php, and so on should have nothing but UnitTestCase classes. No "run test" commands, and no "include('web_test.php');", etc, commands. I'm going back on my earlier statement; we're going to use option 2 from now on, so make sure you put those require()'s back into test-runner.php.

Cheers!
- Nathaniel
Post Reply