Simpletest woes

Discussion of testing theory and practice, including methodologies (such as TDD, BDD, DDD, Agile, XP) and software - anything to do with testing goes here. (Formerly "The Testing Side of Development")

Moderator: General Moderators

josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: Simpletest woes

Post by josh »

Output buffering? I dont see any syntax errors in the code you posted either
User avatar
allspiritseve
DevNet Resident
Posts: 1174
Joined: Thu Mar 06, 2008 8:23 am
Location: Ann Arbor, MI (USA)

Re: Simpletest woes

Post by allspiritseve »

The class under test is missing the first bracket {...

Edit: the problem was due to @include in my __autoload() function.
Last edited by allspiritseve on Tue Jan 20, 2009 9:53 am, edited 1 time in total.
lastcraft
Forum Commoner
Posts: 80
Joined: Sat Jul 12, 2003 10:31 pm
Location: London

Re: Simpletest woes

Post by lastcraft »

Hi..

No, SimpleTest cannot catch certain classes of errors from PHP. Script dead and SimpleTest reduced to collateral damage. If you have the standard error reporting turned off, you get the mystery behaviour above. The favourite one is running out of memory when running in the browser. SimpleTest just appears to stop when you run it's own self test suite.

I started working on the ability to run test cases in their own process, but stopped part way through because I didn't have enough requests for that feature. Got as far as the XML communication, and the DefaultReporter responding to the --xml flag. The last steps is the ability to shell out the test case on to the command line, and to respond properly to an entire test case going AWOL.

Frankly I've wanted this feature for ages and this conversation is a good excuse to finish it. I want to get the next beta release out first though.

yours, Marcus
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Re: Simpletest woes

Post by Chris Corbyn »

Marcus, if you added that feature to simpletest I'd love that. SitePoint have created their own test suite to do this using AJAX, I've created my own test suite to do it with AJAX, or with command line processes and I've definitely seen others do it. It seems that the fact people are going ahead and building their own test suites with this functionality is reason enough to make it a part of the framework :)

Reasons for doing it?

1) Scope: If each test case runs in it's own process you're ensuring a clean environment for each test.
2) Memory: Trying to run 3000 test cases in one process is basically not possible on the standard memory limit.

Gotchas:

If you test data uses multiple character sets (say you're testing i18n stuff) then the generated XML will have character set mismatches and will not be parseable. <![CDATA[ .. ]]> sections do not help because they expect the character character set.

If your tests make assertions regarding binary data, the XML generated is invalid and not parseable by any XML interpreter. We need some way of including binary data in XML. <![CDATA[ .. ]]> sections don't work for some byte values (the NUL byte for one).

The way I got around this way to add assertIdenticalBinary() to my test cases, where pack() is used and then hex strings are compared instead.
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: Simpletest woes

Post by josh »

I dont think ajax is needed, my feature request would be to simply flush the type casting error ( or any non-fatal errors ) to my browser as soon as it (they) occur(s), in case subsequent errors are "fatal" in nature. Honestly I could probably care less about ajax, not to say it wouldn't be "cool" but it doesn't solve any real needs I have currently with the library. I know you cant catch the fatal errors, but I'm saying if there's no fatal errors I get the type casting error caught by simpletest, my request would be to simply patch the html reporter so it was impossible for it to catch errors but have the script error out before they were outputted. Just my 2 cents, but hey if you say ajax is a better way to solve it then do that :D It would be neat if there were just some way to get a "better" error output when you pass NULL to a type casted function like in the example I posted

I appreciate the otherwise wonderful testing framework, very KISS / elegant
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: Simpletest woes

Post by josh »

http://michaelkimsal.com/blog/php5-type-hinting/
I was naive to think its a buffering issue ( realized its not.. ). But check this out.

Edit: nvm I guess this doesn't really help either.. jesus why can't they make it just throw an exception. I'm pulling my hair out here there has to be a way
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: Simpletest woes

Post by josh »

Alright I figured out a workable fix. I implemented a base class, overrode _call and thew an exception about undefined methods. Granted if an object adhere's to an interface this still allows things to seep thru the cracks but thats easier to live with then these darned fatal errors ;-)

Edit: holy crap do my eyes deceive me, I'm getting the type casting error thrown now from simpletest:

Unexpected PHP error [Argument 1 passed to Ne8_Db_Mapper::save() must be an instance of Ne8_Model, instance of Ne8_Collection given, called in C:\Users\Josh\Desktop\ne8\metator\library\Ne8\Db\Mapper.php on line 634 and defined] severity [E_RECOVERABLE_ERROR] in [C:\Users\Josh\Desktop\ne8\metator\library\Ne8\Db\Mapper.php line 394]

WOOP WOOP WOOP

I was lookin into register_shutdown_function and how it exists in a magic "stack" (grrr ) and the same thing w/ fruitless output buffering callbacks. Saw your thread at sitepoint too.
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: Simpletest woes

Post by josh »

Sorry to flood the boards... I'm thinking if I switch to linux xdebug will start working ( known issues with xamp / wamp ) and I'll get stack traces without having to make a base class, which means even if I pass NULL values I can still find out who the caller is
lastcraft
Forum Commoner
Posts: 80
Joined: Sat Jul 12, 2003 10:31 pm
Location: London

Re: Simpletest woes

Post by lastcraft »

Hi all.

Just to let you know I'm still following this thread, but I'm working on the SimpleTest docs at the moment ready for a beta release. I'll try to sneak the testing in a separate process into the beta after that.

I'll also backport it to the 1.0.* branch as I think this would be a cool way of testing mixed PHP4 and PHP5 code bases. Or even multiple versions of PHP so that I could use it for compatibility testing for Simpletest itself.

yours, Marcus
Post Reply