Any point in Unit Testing in this scenario?

Discussion of testing theory and practice, including methodologies (such as TDD, BDD, DDD, Agile, XP) and software - anything to do with testing goes here. (Formerly "The Testing Side of Development")

Moderator: General Moderators

User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Any point in Unit Testing in this scenario?

Post by Chris Corbyn »

I'm digging back up the topic of Unit Testing on my mailer library.

The upshot of the last discussion I had was that Unit Testing just isn't suitable for things like this because you can't guarantee that the server is working correctly or offers the features we need to test.

So... I gave up and left it behind.

But as time goes on it's becoming apparent I need to either take on voluntary code testers who can get a good UAT test coverage, or have another shot at unit testing it because my one man self just can't explore all avenues at each release.

Rather than doing true fine-grained unit testing I'm thinking of just setting up a series of end-to-end tests. Testing the stuff that happens on the PHP side of things is fine, but given the nature of the library I also need to ensure that the email itself is readble, the encoding hasn't screwed up and that attachments work etc. Now that clearly requires an actual person to do that. Does that make the idea of unit testing it sort of pointless?

<side note>
I was looking at PHPUnit rather than SimpleTest but it seems a little sparse. Am I right in thinking simpletest is more featureful?
</side note>
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

Sounds like the right opportunity for unit testing. Specifically, Mock objects. You can fake a real server to interact with. It doesn't need to do any major processing, but it can test the responses your library will send given x, y and z. It's not as realistic as interacting with a real server, but it at least allows you to test responses for a swath of inputs on an island before it goes live.

Also, remember to make unit tests for each "real" bug to make sure they don't get reintroduced accidentally later on.. i.e. regression testing.
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

feyd wrote:Sounds like the right opportunity for unit testing. Specifically, Mock objects. You can fake a real server to interact with. It doesn't need to do any major processing, but it can test the responses your library will send given x, y and z. It's not as realistic as interacting with a real server, but it at least allows you to test responses for a swath of inputs on an island before it goes live.

Also, remember to make unit tests for each "real" bug to make sure they don't get reintroduced accidentally later on.. i.e. regression testing.
I downloaded fakemail for this but in to be blunt about fakemail (no offence to Marcus, it's a nice idea) but it's pretty crap since I can't test ESMTP stuff with it, which basically means I can test sending a mail and that's about all.

My understanding of Mock objects however was not to do with faking external features like a server. I thought stubs sort of did this, but mimmicking a server would be a complete and utter nightmare since it needs to accept socket connections. Not only that, but the amount of code needed to test things TLS or CRAM-MD5 authentication would be more than the library itself and you'd never know if the bug was in the test code or the library on first sight. It seems more reasonable to just use a real server which you can assume will behave as it should.

Just remembered asking a question PHP Code actually about user-defined socket creation and that as asked because I was pondering the fakemail type idea in pure PHP code. I'll see if got any responses (completely forgot I'd posted it).

But basically having a user read an email as the end of the test would be seen as OK, not amateur?

The tests would basically test that the code is doing what we expect and the user would need to make sure the email looks as it should look.
User avatar
Christopher
Site Administrator
Posts: 13596
Joined: Wed Aug 25, 2004 7:54 pm
Location: New York, NY, US

Post by Christopher »

I believe Marcus Baker (lastcraft) and Co. built a mock mail server for doing just this. It is on Sourceforge I believe.
(#10850)
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

arborint wrote:I believe Marcus Baker (lastcraft) and Co. built a mock mail server for doing just this. It is on Sourceforge I believe.
*cough* Fakemail *cough* :lol:

(Read my last post)
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

d11wtq wrote:But basically having a user read an email as the end of the test would be seen as OK, not amateur?

The tests would basically test that the code is doing what we expect and the user would need to make sure the email looks as it should look.
Until we can build better computer vision systems, yes, there is at some point a need for a visual test by a human.. and that's perfectly fine in my book.

I use Unit Tests to take care of the, largely algorithmic, nature of running test code: run these things; any errors? no, continue testing; any errors? yes, okay red flag these and continue if possible.. etc.
User avatar
Ambush Commander
DevNet Master
Posts: 3698
Joined: Mon Oct 25, 2004 9:29 pm
Location: New Jersey, US

Post by Ambush Commander »

there is at some point a need for a visual test by a human..
Precisely. I call them smoke tests (with a perhaps incorrect use of the term), and try to make them as automated as possible. You should have a set of instructions you can follow in order to insure things work properly.

1. Do A
2. Do B
3. Compare result with correct screenshot
User avatar
Christopher
Site Administrator
Posts: 13596
Joined: Wed Aug 25, 2004 7:54 pm
Location: New York, NY, US

Post by Christopher »

d11wtq wrote:*cough* Fakemail *cough* :lol:

(Read my last post)
Missed that, but my assumption was that you would need to extend fakemail to fully test you library -- which would be a good contribution.
(#10850)
nickvd
DevNet Resident
Posts: 1027
Joined: Thu Mar 10, 2005 5:27 pm
Location: Southern Ontario
Contact:

Post by nickvd »

Ambush Commander wrote:
there is at some point a need for a visual test by a human..
Precisely. I call them smoke tests (with a perhaps incorrect use of the term), and try to make them as automated as possible. You should have a set of instructions you can follow in order to insure things work properly.

1. Do A
2. Do B
3. Compare result with correct screenshot
I do believe that you are correct in your use of the term Smoke Test (see: http://en.wikipedia.org/wiki/Smoke_test ... ngineering)

Mozilla Has an extensive suite of smoke tests ( http://www.mozilla.org/quality/smoketests/ ) that need to pass before the release is marked as stable... More or less just instructions and expected output. Think manual unit testing: run function a, does it (equal|not equal) "foo"...

I envision a series of seperate test pages (test1.php, test2.php) that will run both the php tests, and (assuming no emails are actually sent during the tests) do a real run through of the production code, at the bottom of the page would either be a detailed list of required output/results and perhaps screenshots of the email when viewed in the client (or clients if there are potential differences between say hotmail/thunderbird).

I'd me more than happy to help... I have a few different shared hosting accounts (my clients accounts to be precise) that i can run the tests on to get a (slightly) broader range of hardware/versions...
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

I'm writing some fairly broad-scoped tests now. You need to edit a config file to run them. I'll see what you guys think tomorrow since I might go to bed soon (it's 23:30).
arborint wrote:
d11wtq wrote:*cough* Fakemail *cough* :lol:

(Read my last post)
Missed that, but my assumption was that you would need to extend fakemail to fully test you library -- which would be a good contribution.
You know, I did consider having a poke around to make an ESMTP version of Fakemail. Fakemail is more a POC as far as I can make out and it works at what it does, but like I say, it's no real use to me apart from on extremely basic stuff.

It uses Perl's Mail module from CPAN... I'm not overly familiar with that. Been a couple of years since I wrote any perl code too.

Exim is actually very easy to configure to dump the email into a file rather than sending it too. I may have a look at this approach... you just need to write the transport for it.

----

Smoke tests... OK, I'll go with that term :)

My thinking is this:

Generate emails with lipsum type junk in them. Some using a foreign charset such as windows-874 too. Send them out labelled with the subject line linking it to the unit test it's from. The body of the email can then begin by listing (in 7 bit text so it would be virtually impossible to screw up!) things to check in the email and explaining how it should appear.

Let's see what happens. I'm using SimpleTest again...
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

Ack! I hope they've fixed this :(
Notice: Only variable references should be returned by reference in /home/d11wtq/simpletest/simple_test.php on line 74

Notice: Only variable references should be returned by reference in /home/d11wtq/simpletest/simple_test.php on line 62
That's a PHP5 incompatibility issue. I have to admit I have a version of simpletest that's about 2 months out of date so I'll upgrade.

EDIT | This is fixed in 1.01 alpha 3.
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

If the updated 1.0.1beta doesn't work so great, I updated a large portion of my local 1.0.1alpha3 build to work with PHP 5 (with strict on, no less) for Ambush. Hit me on PM if you want it.
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

feyd wrote:If the updated 1.0.1beta doesn't work so great, I updated a large portion of my local 1.0.1alpha3 build to work with PHP 5 (with strict on, no less) for Ambush. Hit me on PM if you want it.
I don't have E_STRICT on, just E_ALL right now. In any case, I run two versions so I'm lucky in that respect since my PHP5 code doesn't have any PHP4 legacy features in it. I'll bear that in mind though and hit you on it if/when I need it ;) Thanks.

Noob Question: Am I safe to assume the tests in a test class will always execute topmost first, down to the bottom-most ? I ask because there's no point duplicating code in different methods if the previous method passed and left things in a state suitable for running the next test.
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

I believe, technically yes, but don't quote me on that. They could be run in any order. That's what the setup() and teardown() methods are for. ;)
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

feyd wrote:That's what the setup() and teardown() methods are for. ;)
Thanks for the pointers. I'll look those up :oops:
Post Reply