Any point in Unit Testing in this scenario?
Moderator: General Moderators
- Chris Corbyn
- Breakbeat Nuttzer
- Posts: 13098
- Joined: Wed Mar 24, 2004 7:57 am
- Location: Melbourne, Australia
Any point in Unit Testing in this scenario?
I'm digging back up the topic of Unit Testing on my mailer library.
The upshot of the last discussion I had was that Unit Testing just isn't suitable for things like this because you can't guarantee that the server is working correctly or offers the features we need to test.
So... I gave up and left it behind.
But as time goes on it's becoming apparent I need to either take on voluntary code testers who can get a good UAT test coverage, or have another shot at unit testing it because my one man self just can't explore all avenues at each release.
Rather than doing true fine-grained unit testing I'm thinking of just setting up a series of end-to-end tests. Testing the stuff that happens on the PHP side of things is fine, but given the nature of the library I also need to ensure that the email itself is readble, the encoding hasn't screwed up and that attachments work etc. Now that clearly requires an actual person to do that. Does that make the idea of unit testing it sort of pointless?
<side note>
I was looking at PHPUnit rather than SimpleTest but it seems a little sparse. Am I right in thinking simpletest is more featureful?
</side note>
The upshot of the last discussion I had was that Unit Testing just isn't suitable for things like this because you can't guarantee that the server is working correctly or offers the features we need to test.
So... I gave up and left it behind.
But as time goes on it's becoming apparent I need to either take on voluntary code testers who can get a good UAT test coverage, or have another shot at unit testing it because my one man self just can't explore all avenues at each release.
Rather than doing true fine-grained unit testing I'm thinking of just setting up a series of end-to-end tests. Testing the stuff that happens on the PHP side of things is fine, but given the nature of the library I also need to ensure that the email itself is readble, the encoding hasn't screwed up and that attachments work etc. Now that clearly requires an actual person to do that. Does that make the idea of unit testing it sort of pointless?
<side note>
I was looking at PHPUnit rather than SimpleTest but it seems a little sparse. Am I right in thinking simpletest is more featureful?
</side note>
- feyd
- Neighborhood Spidermoddy
- Posts: 31559
- Joined: Mon Mar 29, 2004 3:24 pm
- Location: Bothell, Washington, USA
Sounds like the right opportunity for unit testing. Specifically, Mock objects. You can fake a real server to interact with. It doesn't need to do any major processing, but it can test the responses your library will send given x, y and z. It's not as realistic as interacting with a real server, but it at least allows you to test responses for a swath of inputs on an island before it goes live.
Also, remember to make unit tests for each "real" bug to make sure they don't get reintroduced accidentally later on.. i.e. regression testing.
Also, remember to make unit tests for each "real" bug to make sure they don't get reintroduced accidentally later on.. i.e. regression testing.
- Chris Corbyn
- Breakbeat Nuttzer
- Posts: 13098
- Joined: Wed Mar 24, 2004 7:57 am
- Location: Melbourne, Australia
I downloaded fakemail for this but in to be blunt about fakemail (no offence to Marcus, it's a nice idea) but it's pretty crap since I can't test ESMTP stuff with it, which basically means I can test sending a mail and that's about all.feyd wrote:Sounds like the right opportunity for unit testing. Specifically, Mock objects. You can fake a real server to interact with. It doesn't need to do any major processing, but it can test the responses your library will send given x, y and z. It's not as realistic as interacting with a real server, but it at least allows you to test responses for a swath of inputs on an island before it goes live.
Also, remember to make unit tests for each "real" bug to make sure they don't get reintroduced accidentally later on.. i.e. regression testing.
My understanding of Mock objects however was not to do with faking external features like a server. I thought stubs sort of did this, but mimmicking a server would be a complete and utter nightmare since it needs to accept socket connections. Not only that, but the amount of code needed to test things TLS or CRAM-MD5 authentication would be more than the library itself and you'd never know if the bug was in the test code or the library on first sight. It seems more reasonable to just use a real server which you can assume will behave as it should.
Just remembered asking a question PHP Code actually about user-defined socket creation and that as asked because I was pondering the fakemail type idea in pure PHP code. I'll see if got any responses (completely forgot I'd posted it).
But basically having a user read an email as the end of the test would be seen as OK, not amateur?
The tests would basically test that the code is doing what we expect and the user would need to make sure the email looks as it should look.
- Christopher
- Site Administrator
- Posts: 13596
- Joined: Wed Aug 25, 2004 7:54 pm
- Location: New York, NY, US
- Chris Corbyn
- Breakbeat Nuttzer
- Posts: 13098
- Joined: Wed Mar 24, 2004 7:57 am
- Location: Melbourne, Australia
- feyd
- Neighborhood Spidermoddy
- Posts: 31559
- Joined: Mon Mar 29, 2004 3:24 pm
- Location: Bothell, Washington, USA
Until we can build better computer vision systems, yes, there is at some point a need for a visual test by a human.. and that's perfectly fine in my book.d11wtq wrote:But basically having a user read an email as the end of the test would be seen as OK, not amateur?
The tests would basically test that the code is doing what we expect and the user would need to make sure the email looks as it should look.
I use Unit Tests to take care of the, largely algorithmic, nature of running test code: run these things; any errors? no, continue testing; any errors? yes, okay red flag these and continue if possible.. etc.
- Ambush Commander
- DevNet Master
- Posts: 3698
- Joined: Mon Oct 25, 2004 9:29 pm
- Location: New Jersey, US
Precisely. I call them smoke tests (with a perhaps incorrect use of the term), and try to make them as automated as possible. You should have a set of instructions you can follow in order to insure things work properly.there is at some point a need for a visual test by a human..
1. Do A
2. Do B
3. Compare result with correct screenshot
- Christopher
- Site Administrator
- Posts: 13596
- Joined: Wed Aug 25, 2004 7:54 pm
- Location: New York, NY, US
-
nickvd
- DevNet Resident
- Posts: 1027
- Joined: Thu Mar 10, 2005 5:27 pm
- Location: Southern Ontario
- Contact:
I do believe that you are correct in your use of the term Smoke Test (see: http://en.wikipedia.org/wiki/Smoke_test ... ngineering)Ambush Commander wrote:Precisely. I call them smoke tests (with a perhaps incorrect use of the term), and try to make them as automated as possible. You should have a set of instructions you can follow in order to insure things work properly.there is at some point a need for a visual test by a human..
1. Do A
2. Do B
3. Compare result with correct screenshot
Mozilla Has an extensive suite of smoke tests ( http://www.mozilla.org/quality/smoketests/ ) that need to pass before the release is marked as stable... More or less just instructions and expected output. Think manual unit testing: run function a, does it (equal|not equal) "foo"...
I envision a series of seperate test pages (test1.php, test2.php) that will run both the php tests, and (assuming no emails are actually sent during the tests) do a real run through of the production code, at the bottom of the page would either be a detailed list of required output/results and perhaps screenshots of the email when viewed in the client (or clients if there are potential differences between say hotmail/thunderbird).
I'd me more than happy to help... I have a few different shared hosting accounts (my clients accounts to be precise) that i can run the tests on to get a (slightly) broader range of hardware/versions...
- Chris Corbyn
- Breakbeat Nuttzer
- Posts: 13098
- Joined: Wed Mar 24, 2004 7:57 am
- Location: Melbourne, Australia
I'm writing some fairly broad-scoped tests now. You need to edit a config file to run them. I'll see what you guys think tomorrow since I might go to bed soon (it's 23:30).
It uses Perl's Mail module from CPAN... I'm not overly familiar with that. Been a couple of years since I wrote any perl code too.
Exim is actually very easy to configure to dump the email into a file rather than sending it too. I may have a look at this approach... you just need to write the transport for it.
----
Smoke tests... OK, I'll go with that term
My thinking is this:
Generate emails with lipsum type junk in them. Some using a foreign charset such as windows-874 too. Send them out labelled with the subject line linking it to the unit test it's from. The body of the email can then begin by listing (in 7 bit text so it would be virtually impossible to screw up!) things to check in the email and explaining how it should appear.
Let's see what happens. I'm using SimpleTest again...
You know, I did consider having a poke around to make an ESMTP version of Fakemail. Fakemail is more a POC as far as I can make out and it works at what it does, but like I say, it's no real use to me apart from on extremely basic stuff.arborint wrote:Missed that, but my assumption was that you would need to extend fakemail to fully test you library -- which would be a good contribution.d11wtq wrote:*cough* Fakemail *cough*
(Read my last post)
It uses Perl's Mail module from CPAN... I'm not overly familiar with that. Been a couple of years since I wrote any perl code too.
Exim is actually very easy to configure to dump the email into a file rather than sending it too. I may have a look at this approach... you just need to write the transport for it.
----
Smoke tests... OK, I'll go with that term
My thinking is this:
Generate emails with lipsum type junk in them. Some using a foreign charset such as windows-874 too. Send them out labelled with the subject line linking it to the unit test it's from. The body of the email can then begin by listing (in 7 bit text so it would be virtually impossible to screw up!) things to check in the email and explaining how it should appear.
Let's see what happens. I'm using SimpleTest again...
- Chris Corbyn
- Breakbeat Nuttzer
- Posts: 13098
- Joined: Wed Mar 24, 2004 7:57 am
- Location: Melbourne, Australia
Ack! I hope they've fixed this 
EDIT | This is fixed in 1.01 alpha 3.
That's a PHP5 incompatibility issue. I have to admit I have a version of simpletest that's about 2 months out of date so I'll upgrade.Notice: Only variable references should be returned by reference in /home/d11wtq/simpletest/simple_test.php on line 74
Notice: Only variable references should be returned by reference in /home/d11wtq/simpletest/simple_test.php on line 62
EDIT | This is fixed in 1.01 alpha 3.
- Chris Corbyn
- Breakbeat Nuttzer
- Posts: 13098
- Joined: Wed Mar 24, 2004 7:57 am
- Location: Melbourne, Australia
I don't have E_STRICT on, just E_ALL right now. In any case, I run two versions so I'm lucky in that respect since my PHP5 code doesn't have any PHP4 legacy features in it. I'll bear that in mind though and hit you on it if/when I need itfeyd wrote:If the updated 1.0.1beta doesn't work so great, I updated a large portion of my local 1.0.1alpha3 build to work with PHP 5 (with strict on, no less) for Ambush. Hit me on PM if you want it.
Noob Question: Am I safe to assume the tests in a test class will always execute topmost first, down to the bottom-most ? I ask because there's no point duplicating code in different methods if the previous method passed and left things in a state suitable for running the next test.
- Chris Corbyn
- Breakbeat Nuttzer
- Posts: 13098
- Joined: Wed Mar 24, 2004 7:57 am
- Location: Melbourne, Australia