Page 1 of 2
Suite Tester 1.0 Release Candidate
Posted: Thu May 03, 2007 8:25 pm
by Nathaniel
For all of us SimpleTest users, I've got a Suite Tester release candidate up. I've been using the 0.4 version myself for half of a year, and it's been incredibly helpful. I didn't have much of a workload this past week so I decided to recode it from scratch.
I've put a page about
Suite Tester with some demos, so if you have a spare half-hour over the weekend, I'd appreciate it if you'd give it a go and let me know what you think

Once I have some initial community feedback I'd like to publicize it a bit more.
Thanks,
Nathaniel
Posted: Fri May 04, 2007 4:45 am
by Maugrim_The_Reaper
You bought me with the first line on the web page...
Conveniently test an entire application or a single unit in one place!
I didn't quite follow the introduction at first. Am I right in saying it tests the entire suite at the start, and then updates individual tests as you direct? That by itself is a time saver...
- If you add Group folding (let me hide those I already know are passing) it would make navigation easier. I have test suites with a lot more than 7 test cases. Scrolling down to click a test case, and then back up to see the results will get tiresome eventually

.
- Also it would be nice for an entire Group to be capable of a rerun.
- Optional display of a test's detailed output by running it in a popup/floating-div (a separate optional link alongside the current test title links maybe). Not as important as the main summary view but it's output I often go looking for.
- Is $_GET['test'] sanitised before hitting the include() statement? The statement also has the @ error suppressor for some reason.
- May need to adapt to super-long test names without breaking the page width.
- I'd like to see more obvious highlighting of failing test cases - like the red bar effect instead of just the red text. I know you have a CSS file in the package so it's probably just a minor point.
Nice going!
It looks like a really cool tool.
Posted: Fri May 04, 2007 7:59 am
by Nathaniel
Hey Maugrim!
Maugrim_The_Reaper wrote:You bought me with the first line on the web page...

I didn't quite follow the introduction at first. Am I right in saying it tests the entire suite at the start, and then updates individual tests as you direct?
Yup. Of course, if you want to reload every test, just hit F5. A "Reload All" button is planned for the next version.
If you add Group folding (let me hide those I already know are passing) it would make navigation easier. I have test suites with a lot more than 7 test cases. Scrolling down to click a test case, and then back up to see the results will get tiresome eventually

.
Ooh, that's a great idea! Would you make it fold automatically if the group passes, then? With a "toggle folding" link next to it?
Also it would be nice for an entire Group to be capable of a rerun.
Another good idea. Hopefully it'll make it into the next version.
Optional display of a test's detailed output by running it in a popup/floating-div (a separate optional link alongside the current test title links maybe). Not as important as the main summary view but it's output I often go looking for.
I'm not quite getting this one. Do you want this extra link to spit out all of the passing tests? That could be useful.
Is $_GET['test'] sanitised before hitting the include() statement? The statement also has the @ error suppressor for some reason.
Winces no... I usually have all of my testing software locked in a password-protected directory and the thought didn't occur to me.
As for the @, I think I added that because I didn't like seeing a page full of include errors when a test file was missing. I'll take it out though. Thanks for finding that.
May need to adapt to super-long test names without breaking the page width.
Hmm, I don't quite understand. Like make the test name "A Test of ... Model"? I guess I'll have to populate a suite and try to replicate the problem you had.
I'd like to see more obvious highlighting of failing test cases - like the red bar effect instead of just the red text. I know you have a CSS file in the package so it's probably just a minor point.
Ok. Easily enough done.
Is this what you were looking for? That'll make it into the 1.0 release; no need to wait for 2.0 there.
Nice going! It looks like a really cool tool.
Thanks, and thanks for the feedback

Posted: Fri May 04, 2007 10:39 am
by Nathaniel
I have updated the
Suite Tester page with Release Candidate 2. The
Version History details the changes.
Posted: Fri May 04, 2007 11:43 am
by RobertGonzalez
That is pretty neat Nathaniel. I think I am going to be playing with this muchly over the next few days.
Posted: Fri May 04, 2007 4:11 pm
by Ambush Commander
It looks interesting. I personally hacked up a little test-suite selector for my own project, but if Suite Tester offers more benefits I'll use it!
Question: is there an AJAX way to refresh the entire test suite, or do I have to reload? Also, is there a way to continually refresh a test result every, say, two seconds? With that, I wouldn't have to continually switch to my browser and click refresh every time I want to test a code change! The flickering that happens when you load reload a test is slightly annoying, but not major.
Posted: Fri May 04, 2007 4:30 pm
by Nathaniel
Ambush Commander wrote:Question: is there an AJAX way to refresh the entire test suite, or do I have to reload?
Currently we have to reload. A "Refresh All" button is in the feature queue, however.
Also, is there a way to continually refresh a test result every, say, two seconds?
Interesting idea. Prototype has a "PeriodicalUpdater" class built into it, so it shouldn't be hard to implement.
What about "on window focus, refresh the last test run"? Firebug shows the test cases taking about 60ms to load, so lag time would be negligent.
The flickering that happens when you load reload a test is slightly annoying, but not major.
Do you mean when you f5 the page, or when you click an individual test case to be rerun?
Posted: Fri May 04, 2007 4:35 pm
by Ambush Commander
What about "on window focus, refresh the last test run"? Firebug shows the test cases taking about 60ms to load, so lag time would be negligent.
That sounds good, although, you probably want to give the user more control over which tests to refresh.
Do you mean when you f5 the page, or when you click an individual test case to be rerun?
When individual test-cases get rerun. The associated messages get erased, which often moves the menu widget around: the menu of test cases should stay in the same position.
Posted: Sun May 06, 2007 4:38 pm
by sike
looks good nathaniel (:
as most other unit testers i have whipped up my own test selector featuring a hierarchical view and a basic code coverage display. it would be nice if you could incorporate these features because then i would definitely switch (:
chris
ps. i have put a little demo on
http://code.refactored.de/test/external ... /index.php
Posted: Mon May 07, 2007 2:42 am
by matthijs
Ok, I have a running simpletest lib, and get the suite-tester running, but I only get "failed, with no response" results. While my simpletest results are definitely green.
My current setup is:
/suite-tester/
/tests/all-tests.php
/tests/login_test.php
.. etc
/tests/simpletest/
Now I should edit the correct paths in suite.js. But whichever path I choose, it doesn't find my tests. What should I change in my setup?
Posted: Mon May 07, 2007 8:39 am
by Nathaniel
Hi matthijs,
The "failed, with no response" error means that library/test-runner.php is returning a blank file.
The most probable cause for this is because it's not reporting errors - probably the error generated when it can't find the file you want it to test.
Would you please open library/test-runner.php, find line 14, and insert this code?
Code: Select all
error_reporting(E_ALL);
ini_set('display_errors', '1');
Then try running Suite Tester again.
Also, can you tell me what version of PHP you are using? I haven't tested it on PHP 5, now that I think about it.
Hey Chris, glad you like it. I need to go at the moment, but I'll reply to you in a bit. Thanks for the comment!
- Nathaniel[/syntax]
Posted: Mon May 07, 2007 9:17 am
by matthijs
Ok, I added the error reporting, and, after changing the path again, get a succesful running of the examples, but an error for my own tests:
Myclimbing Test Case failed. Response Text:
Fatal error: Cannot redeclare class fieldexpectation in /Applications/MAMP/htdocs/tests/simpletest/web_tester.php on line 23
3/3 test cases complete: 2 passes, 0 fails, and 1 exceptions.
I'm running php 5.1
Posted: Mon May 07, 2007 9:32 am
by Nathaniel
Ok, so your test files are include()ing the simpletest files. If you look down in test-runner.php, I require all the files you should need for you.
Two options:
1. remove these lines from test-runner.php:
Code: Select all
require(SIMPLETEST_DIR . 'unit_tester.php');
require(SIMPLETEST_DIR . 'web_tester.php');
require(SIMPLETEST_DIR . 'reporter.php');
require(SIMPLETEST_DIR . 'mock_objects.php');
2. Remove the require()/include() calls from your unit tests.
Cheers,
- Nathaniel
Posted: Mon May 07, 2007 10:18 am
by matthijs
Ok, thanks.
Option 1:
Example Test Case failed. Response Text:
Fatal error: Class 'TestSuite' not found in /Applications/MAMP/htdocs/suite-tester/library/test-runner.class.php on line 8
Myclimbing Test Case failed. Response Text:
Fatal error: Class 'TestSuite' not found in /Applications/MAMP/htdocs/suite-tester/library/test-runner.class.php on line 8
A Test Inside a Group failed. Response Text:
Fatal error: Class 'TestSuite' not found in /Applications/MAMP/htdocs/suite-tester/library/test-runner.class.php on line 8
3/3 test cases complete: 0 passes, 0 fails, and 3 exceptions
Option 2:
Myclimbing Test Case failed. Response Text:
Warning: TestSuite::include(web_test.php) [function.TestSuite-include]: failed to open stream: No such file or directory in /Applications/MAMP/htdocs/tests/simpletest/test_case.php on line 461
Warning: TestSuite::include() [function.include]: Failed opening 'web_test.php' for inclusion (include_path='.:/Applications/MAMP/bin/php5/lib/php') in /Applications/MAMP/htdocs/tests/simpletest/test_case.php on line 461
All tests
Fail: web_test.php -> Bad TestSuite [web_test.php] with error [Failed opening 'web_test.php' for inclusion (include_path='.:/Applications/MAMP/bin/php5/lib/php')]
0/0 test cases complete: 0 passes, 1 fails and 0 exceptions.
Warning: Cannot modify header information - headers already sent by (output started at /Applications/MAMP/htdocs/tests/simpletest/test_case.php:461) in /Applications/MAMP/htdocs/suite-tester/library/json-reporter.php on line 40
{ "messages" : [ { "type" : "exception", "msg" : "../../tests/all_tests.php -> Bad TestSuite: ../../tests/all_tests.php has no runnable test cases." } ], "passes" : "0" }
3/3 test cases complete: 2 passes, 0 fails, and 1 exceptions.
Maybe we could solve this if you'd tell me how the setup should look like. As my simpletest is setup now:
/htdocs/
/htdocs/suite-tester/
/htdocs/tests/
/htdocs/tests/all_tests.php
/htdocs/tests/log_tests.php
/htdocs/tests/login_tests.php
...
/htdocs/tests/simpletest/
the tests works fine if I run them directly by calling /tests/all_tests.php
(I have been working with PHP for a while now, but somehow I have a love-Hate relationship with includes and requires. It never goes well...)
Posted: Mon May 07, 2007 11:42 am
by Nathaniel
The way you have the hierarchy set up shouldn't matter so long as everything points to each other correctly.
Could you pm me a link to your site so I can take a peek at stuff?
If not: It sounds like you're running the tests in all_tests.php (etc). all_tests.php, log_tests.php, and so on should have nothing but UnitTestCase classes. No "run test" commands, and no "include('web_test.php');", etc, commands. I'm going back on my earlier statement; we're going to use option 2 from now on, so make sure you put those require()'s back into test-runner.php.
Cheers!
- Nathaniel