Page 2 of 3

Re: Testing the output

Posted: Tue Jul 06, 2010 12:04 pm
by McGruff
josh wrote:If the whole site is down that isn't going to slip past my manual testers. Surely someone as experienced as yourself knows you can't just stop manual tests, no matter how strong your automated tests are. To me it seems like you're making the argument we all HAVE to do things your way. I'm saying I chose not to.
Manual testing is just not good enough.

For starters it's inefficient. Presumably these are billable hours? Computers don't - yet - ask to be paid for their work.

A computer is more accurate. On the site I mentioned, acceptance tests make over a thousand individual assertions. Would manual testers remember to check everything, every time? Not a chance. Machines never make mistakes - unless you tell them to.

A computer is quicker. With automated tests I pretty much get instant feedback at the click of a button. Well OK, not that instant: the unit tests are fast but the acceptance tests take about five minutes. However, that's still fast enough to run them regularly, several times a day, while I'm working. If I'm being very generous it would take ten times longer to do the same tests manually, clicking around in a browser. You'd be lucky to get a full test run once per day and that has a very significant cost. Any new bugs which you introduce won't be picked up quickly and by the time you do find them they'll be harder to fix. There's more code to hunt through and it's no longer fresh in your mind. It's much, much easier to stay on top of things if you take lots of little steps, each verified by a test run.

It's not about "my way" :roll: it's about learning to do the best you can. I didn't invent agile programming but I have tried it, or at least some aspects of it, and it works. This isn't just some academic theory. There are real, practical benefits. I did actually cut a few corners on the current job. There were the usual pressures of deadlines to be met. However, I now have new features to add and I'm scared to change anything because there are holes in the acceptance test coverage. We've now got hundreds of paying customers and I cannot allow any mistakes onto the live site. Not one. So the new stuff is being held up because I have to go back and do what I should have done properly in the first place.

I don't think I've ever skimped on testing without regretting it later.

Re: Testing the output

Posted: Tue Jul 06, 2010 2:11 pm
by josh
I don't think anyone is saying tests are a bad thing. You must not have read my post:
Josh wrote:Brittle is just a word I'm not criticizing it just characterizing it, since they produce a high rate of test failures, you "harden" your tests, but you add a ton of work for yourself basically. Browser based tests are an asset, but I prefer the defect localization of a unit test.
If you think writing tests is free, come create some test suites for some legacy projects I have to work on. Tests aren't free, they are in investment that has a cost. It would be stupid to make an investment if you think the cost is more than the ROI. I never said all investing is bad ;-)

Even if you have the best tests in the world, if you skip your manual tests you're going to screw up. You're going to forget to write a test or some HTML will look wacky to the human but fine to the computer. In my opinion if you don't do manual testing you're an irresponsible developer. When we test we test things down the pixel. We have decided as a team that automated tests cannot tell us if something is 1px off due to a CSS glitch, or what have you...

So we write test for the time consuming parts, filling out forms, validating forms, those are my unit tests. When I do manual testing, I just make sure the form is showing up. I unit test all my controllers, all my templates, and then I manually test it all working together, from within each browser.

Re: Testing the output

Posted: Tue Jul 06, 2010 6:41 pm
by McGruff
josh wrote:I don't think anyone is saying tests are a bad thing. You must not have read my post:
Josh wrote:Brittle is just a word I'm not criticizing it just characterizing it, since they produce a high rate of test failures, you "harden" your tests, but you add a ton of work for yourself basically. Browser based tests are an asset, but I prefer the defect localization of a unit test.
Not sure what I missed? You seemed to be saying browser-based tests aren't very useful because they're brittle? I would argue they are essential and I don't understand why you find them brittle.
josh wrote:If you think writing tests is free, come create some test suites for some legacy projects I have to work on. Tests aren't free, they are in investment that has a cost. It would be stupid to make an investment if you think the cost is more than the ROI. I never said all investing is bad ;-)
Writing tests after the fact is a real chore - agreed. Testing makes a lot more sense if you code test-first - and of course then it's about much more than just testing ;) Personally I'd refuse to work on an app like that. Programming is just too difficult without a proper test suite.
josh wrote:Even if you have the best tests in the world, if you skip your manual tests you're going to screw up. You're going to forget to write a test or some HTML will look wacky to the human but fine to the computer. In my opinion if you don't do manual testing you're an irresponsible developer. When we test we test things down the pixel. We have decided as a team that automated tests cannot tell us if something is 1px off due to a CSS glitch, or what have you...
OK you can't test graphic design. You will need to flick through the site to try it out on different browsers (except for IE: we can just assume it will f***k things up). There's nothing you can do about that. But every other kind of manual test, the ones which check content rather than presentation, can and should be automated.

Re: Testing the output

Posted: Tue Jul 06, 2010 9:39 pm
by josh
McGruff wrote:You seemed to be saying browser-based tests aren't very useful because they're brittle?
They are extremely useful. What I said is they have a higher cost of maintenance (I used the term brittle) and you can test your output with a regular unit test. Just because you're testing output doesn't automatically mean you want a browser based test. Consider various kinds of tests before jumping in.

Re: Testing the output

Posted: Wed Jul 07, 2010 12:48 am
by McGruff
josh wrote:they have a higher cost of maintenance (I used the term brittle)
I don't understand what you mean. Maybe you're over-specifying in your tests?
josh wrote:and you can test your output with a regular unit test. Just because you're testing output doesn't automatically mean you want a browser based test. Consider various kinds of tests before jumping in.
Unit tests don't test browser output. They test components which are hidden away under the bonnet.

Acceptance tests are a completely different kind of test. Here we're manipulating the application interface in the same way as a real life user. Links are clicked, forms are filled in and assertions are made about page contents. The application is being tested as a whole, not the individual components in isolation.

Re: Testing the output

Posted: Wed Jul 07, 2010 10:53 am
by josh
Are you asking clarification about the way I do things?
Unit tests don't test browser output. They test components which are hidden away under the bonnet.
With a unit test I'd test that the __toString method of a CMS page "$page" object properly renders some HTML snippet. There would be no concept of http request in my unit test (since its a unit test). Am I testing browser output? Of course not. I am testing output though.

That way if the designer comes in and renames a CSS class name, I don't have to adjust any tests. We have a team member check those pages manually for continuity anyways. My unit test ensures that CMS Page properly renders itself, and later on a human ensures that CMS page is actually showing up in the application and seems to be responding properly, she makes sure it looks visually good from a human standpoint as well. Works well for us. :)

Re: Testing the output

Posted: Wed Jul 07, 2010 2:24 pm
by McGruff
The OP:
I want to make sure the page works
This needs acceptance tests ie tests which interact with an installed application just like an ordinary user and make assertions about page content. That's why I suggested the SimpleTest webtester. It's the simplest and best way to do this. That's just how it's done.

I'm not clear what you're trying to say but your unit tests do not do the same job as acceptance tests and your manual testing could (and should) be replaced with automated, acceptance tests (with one or two exceptions - OK you can't test graphic design). Manual testing is not just another option and you can pick whichever you prefer. It doesn't get the job done. That might be OK for a non-critical application like this forum but it certainly isn't for something like ecommerce.

Re: Testing the output

Posted: Wed Jul 07, 2010 5:19 pm
by josh
You're right - having unit tests for components that generate output is "wrong". My experiences aren't relevant to anybody. I obviously answered the topic wrong and am an idiot. Sarcasm. All I did was state how I do things and state that they worked well. I shared my experiences. I'm sorry that my experiences contradicts your beliefs. Don't know what else to say.

I never said that all your tests should be manual. I just said that you cannot automate every test, you will always do manual testing. If you disagree then you suck at math. (There are an infinite number of code paths & interactions of features in a complex program.) If you think you covered every scenario then you are over-confident in your tests. That's why I exercise moderation in writing them.

Re: Testing the output

Posted: Wed Jul 07, 2010 7:52 pm
by McGruff
It's not personal. It's about learning to program as best we can. I would like everyone who might be reading this to learn why automated tests are important. That's just my opinion of course except that I've got enough experience to know damn well that manual testing won't work. Sorry but it's got to be said. I'm not going to fudge anything just because someone might get upset.

To the OP: check out the advanced php forum on sitepoint if you're interested in trying out SimpleTest and web testing. There are a few test-addicts to be found there and you might even get to chat with the author, Marcus Baker.

Re: Testing the output

Posted: Wed Jul 07, 2010 9:44 pm
by josh
McGruff wrote:automated tests are important.
I agree so quit the straw man arguing and drop it. :banghead: You're misrepresenting my position on automated tests & browser tests, making it out to seem like I am not an advocate when I am, just because I also pointed to manual tests and said they are also useful. So yes its personal. You fail to understand that I advocate all forms of testing, so can we please let this die now. I bet the OP could care less at this point, you told him 5x ;-) I only reply to correct the record on my stance.

Re: Testing the output

Posted: Wed Jul 07, 2010 10:29 pm
by McGruff
You did say this:
I unit test all my controllers, all my templates, and then I manually test it all working together, from within each browser.
Straw men indeed.

Re: Testing the output

Posted: Thu Jul 08, 2010 11:41 am
by josh
Right. I wrote how I test on one project. I didn't say how you should test on your project. I specifically said browser based tests are an asset. You know what an asset means right? It means I think they are very valuable.

Re: Testing the output

Posted: Sun Jul 11, 2010 11:02 am
by Bruno De Barros
God damn, you guys have gotten way too carried away. :P In my opinion manual testing is always necessary. The less manual testing you have to do the better, of course, since, as McGruff pointed, manual testing can be error-prone. But if I have a template for a form, and I expect it to be filled with certain values by the time it gets to the browser, I can make a test that checks that the processed template (the form, filled with said certain values) can be found in the output of the script. No need to test on the browser, or anything. I'll be sure it works, because the form's HTML was just fine when I coded the template. Granted, I could have made a HTML mistake while coding the template, but that's where Simpletest's webtester AND manual testing are necessary, to make sure it -really- works.

But in my opinion, after the HTML is coded and you know that it works (that there's no glitches or anything in any browser), you can fill it up with content, and test whether the output is what was expected (for example, testing that after you enter a wrong username and password, the HTML error message div you coded will show up, that the form's username field is repopulated, and all of that).

I mean, for most projects I've worked on, my problem isn't making sure the HTML works in every browser and all of that. It's making sure that my PHP code is populating my HTML templates the way I intend it to, and that's where I think I could use normal unit tests for verification. If the HTML works, and PHP is populating it with the expected content, then it should work as intended.

Re: Testing the output

Posted: Mon Jul 12, 2010 3:24 pm
by josh
Bruno De Barros wrote:I mean, for most projects I've worked on, my problem isn't making sure the HTML works in every browser and all of that. It's making sure that my PHP code is populating my HTML templates the way I intend it to, and that's where I think I could use normal unit tests for verification. If the HTML works, and PHP is populating it with the expected content, then it should work as intended.
Yeah you still don't have to go thru the browser to "document" (test) that. I can unit test a template & do it quite often when writing plugins for 3rd party systems. Turn on output buffering, include the controller and/or template, and assert against the output buffer or response object. It may or may not be a good idea, but its possible because I do it everyday.

If you prefer browser tests then use browser tests. If you're like me and don't prefer them, you have other options besides manual tests. I guess part of the dichotomy in McGruff and my positions is that I think TDD is not the same thing as automated tests. An automated tester has the philosophy that automated tests can automate everything. A TDD developer has the philosophy that you're documenting how the code should work, not testing it. In practice you want to be somewhere in between the two in my opinion. You want to automate the stuff that is truly cumbersome to manually test for sure.

Re: Testing the output

Posted: Mon Jul 12, 2010 4:01 pm
by Bruno De Barros
Tests can't automate everything, obviously, but I find it that if I have a load of pages to manually test, and a load of behaviors to test... Well, I'm bound to forget some of them, especially if I'm trying to finish up a project and the deadline's getting closer. I mean, I've been manually testing the websites until now, and the only time I use unit tests is when I'm developing libraries/classes, and I just wanted to automate more of my testing. I read an article the other day on zend's devzone about using selenium and phpunit for acceptance testing, so I'm hoping that will help me out.