Getting started with Unit Testing

Discussion of testing theory and practice, including methodologies (such as TDD, BDD, DDD, Agile, XP) and software - anything to do with testing goes here. (Formerly "The Testing Side of Development")

Moderator: General Moderators

Post Reply
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Getting started with Unit Testing

Post by Chris Corbyn »

I've decided I'm gonna test Swift inside out, upside down and back-to-front. I'm completely new to Unit testing so I figured I'd learn by doing it on something useful.

I've decided to go with SimpleTest and after reading the docs partway through I got impatient and though bah I'll just start and see what happens (I was finding the docs a little tedious to be honest).

I figured the best place to start would be at the heart of Swift and test the connection object (SMTP first).

I've only written one test so far and hopefully, with just a few adjustments it should apply to PHP5 too (loose the interface).

The test looks a bit long and repetitive though. Am I testing too many factors that are just, well, obvious?

Code: Select all

<?php

set_time_limit(0);

require_once('../docs/Swift_IConnection.php');
require_once('../Swift/Swift_SMTP_Connection.php');
require_once('../../simpletest/unit_tester.php');
require_once('../../simpletest/reporter.php');

class TestOfSMTPConnection extends UnitTestCase
{
	public function TestCreatingUnencryptedConnectionOnDefaultPort()
	{
		$connection = new Swift_SMTP_Connection('smtp.gmail.com', SWIFT_DEFAULT_PORT);
		$this->assertFalse($connection->isConnected());
		$this->assertFalse($connection->readHook);
		$this->assertFalse($connection->writeHook);
		$this->assertTrue($connection->start());
		$this->assertTrue($connection->isConnected());
		$this->assertTrue($connection->readHook);
		$this->assertTrue($connection->writeHook);
		unset($connection);
		$connection = new Swift_SMTP_Connection('smtp.gmail.com');
		$this->assertFalse($connection->isConnected());
		$this->assertFalse($connection->readHook);
		$this->assertFalse($connection->writeHook);
		$this->assertTrue($connection->start());
		$this->assertTrue($connection->isConnected());
		$this->assertTrue($connection->readHook);
		$this->assertTrue($connection->writeHook);
		unset($connection);
	}
}

$test =& new TestOfSMTPConnection;
$test->run(new HtmlReporter());

?>
For example, I've tested it with the constant, and then without it even though the constant is actually coded into the constructor as the default value. It's also pretty obvious that the handles will be false if nothin has been done in the object.

Maybe this is more sensible?

Code: Select all

<?php

set_time_limit(0);

require_once('../docs/Swift_IConnection.php');
require_once('../Swift/Swift_SMTP_Connection.php');
require_once('../../simpletest/unit_tester.php');
require_once('../../simpletest/reporter.php');

class TestOfSMTPConnection extends UnitTestCase
{
	public function TestCreatingUnencryptedConnectionOnDefaultPort()
	{
		$connection = new Swift_SMTP_Connection('smtp.gmail.com', SWIFT_DEFAULT_PORT);
		$this->assertFalse($connection->isConnected());
		$this->assertTrue($connection->start());
		$this->assertTrue($connection->isConnected());
		$this->assertTrue($connection->readHook);
		$this->assertTrue($connection->writeHook);
		unset($connection);
	}
}

$test =& new TestOfSMTPConnection;
$test->run(new HtmlReporter());

?>
If I write all my tests like I wrote the first one then I'm going to have a good amount more code in the tests than I do in Swift itself :?

Just looking for a "carry on..." :) before I go too far doing the wrong thing.

Cheers,

d11
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

with thorough testing, you can often end up with a lot of code in the test. Since your class is, on the whole, fairly simple, it's quite possible your tests will have more code.. until you start finding places to refactor and stuff, thanks to the tests.. ;)
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

feyd wrote:with thorough testing, you can often end up with a lot of code in the test. Since your class is, on the whole, fairly simple, it's quite possible your tests will have more code.. until you start finding places to refactor and stuff, thanks to the tests.. ;)
Cheers. I did find some ways to improve that test by adding type checks in there. Mainly for the handles. TRUE by itself doesn't tell me alot, I need to know that they make is_resource() return ture ;) I could even go so far as attempting to write/read from them but I think that's not for these tests, and probably for testing Swift's read/write methods itself, unless testing that at this point enforces why it SHOULD work from within Swift :? Hmm..
User avatar
Maugrim_The_Reaper
DevNet Master
Posts: 2704
Joined: Tue Nov 02, 2004 5:43 am
Location: Ireland

Post by Maugrim_The_Reaper »

I was about to unit test your lib over the weekend but I got distracted by the actual appearance of real sunlight in Ireland...:).

First of all I'd suggest you organise the test calls into a central file... I use something like:

Code: Select all

<?php

// keep the code as clean as possible!
error_reporting(E_ALL);

// Exact same path as Partholan's APPRROOT
define('APPROOT', realpath('..') . DIRECTORY_SEPARATOR);
// Path to this directory /Tests
define('TESTROOT', dirname(__FILE__) . DIRECTORY_SEPARATOR);

require_once(TESTROOT . 'simpletest/unit_tester.php');
require_once(TESTROOT . 'simpletest/mock_objects.php');
require_once(TESTROOT . 'simpletest/reporter.php');

$test = new GroupTest('GroupTest: Partholan');

// Tests are automatically run for each class in the following files
$test->addTestFile(TESTROOT . 'Partholan/Test_ServiceLocator.php');
$test->addTestFile(TESTROOT . 'Partholan/Test_ActionRequestMapper.php');
$test->addTestFile(TESTROOT . 'Partholan/Test_DataObject.php');
$test->addTestFile(TESTROOT . 'Partholan/Test_DataAccess.php');
$test->addTestFile(TESTROOT . 'Partholan/Test_Request.php');

// Tests can operate from the command line or through a browser
if(SimpleReporter::inCli())
{
	exit ($test->run(new TextReporter()) ? 0 : 1);
}
$test->run(new HtmlReporter());

?>
Next up, consider what to test. Sometimes there are checks which do not require testing in-depth. I usually skip testing the initial values of variables for example (it's obvious in the class), getters, setters, and other no-brainer simple stuff. This reduces the coverage of the tests - which at the end of the day just means less tests to write. Relating to your tests - do you really need to test all those False assertions?

Another thing to consider - can I run your tests offline? If not, I will encounter numerous errors while offline and be unable to modify Swift without disabling/swapping out tests. I think there's an SMTP resource out there for unit testing, also handy for testing classes which result in emails that you need to test. (Edit: http://www.lastcraft.com/fakemail.php )

Also split individual tests into separate test methods. Makes error discovery and tracking that much simpler.

Now onto the Tests (and keep in mind I'm a relative newbie too). Your tests seem to make a few small mistakes. First of all keep in mind your test code is still PHP - all those PHP constructs you know can be used here to improve the tests. For example, there is nothing wrong in testing all constant values - they are part of the public interface. But how about lumping each into an array, and re-running the same block of tests for each iteration? Just replace the constants with their expected values maybe. Then start(), assertions, stop(), etc.

You also have class variables exposed publicly - are all the class variables intended to be publicly exposed? Are some primarily private? This may not apply, but if you have a private variable in a class that you need to test, you may need some "loose wiring". This is when you have a class you know works, but can't test it on the spot - you need to add extra methods (or just sub class it) to facilitate testing. Private variables you must test are one common sign of this in PHP5. E.g. In my ServiceLocator test I set a searchLocation private array - to check it's correct I need to access it publicly (but it's private and disallowed). Solution: Add a public getter, or sub-class the class for the test and override searchLocation to make it public.

What about stop()? This seems to be a public method to close the connection. Should it be tested?

Would something like this be possible?? Maybe add a PHP command to setup Fakemail for each port? (Not sure if possible).

Code: Select all

$constant_value_array = array(
	0,
	1,
	2,
	25,
	465
}
foreach($constant_value_array as $swift_constant_value)
{
	$connection = new Swift_SMTP_Connection('smtp.gmail.com', $swift_constant_value);
	$this->assertTrue($connection->start());
	$this->assertTrue($connection->isConnected());
	$this->assertTrue($connection->readHook);
	$this->assertTrue($connection->writeHook);
	// add assertions if stop() called (if any)
}
Where is Swift_IConnection required?

Hope the above is of some help![/quote]
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

Thanks Maugrim. Swift_IConnection is actually hard-coded into thr Swift.php file for general use, but there's a separate file in docs for development and testing purposes. Putting the interface in the main file seemed like a good idea at the time since it made it easier for general use but I've hit the first problem in testing because of that.

all_tests.php might try to include an interface even though it's already been declared in Swift.php... I've had to add in checks to make sure the interface doesn't exist and the Swift class doesn't exist before requiring the interfaces.

Ok now I have a confession. I've duplicated this thread over at SitePoint. Someone there mentioned that this seems more suited to Regression testing since it communicates with the outside world. Would anybody be any the wiser on this?

FakeMail was mentioned by Marcus so I'll definitely have a look at that :)
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

I've just installed fakemail but I see a few caveats.

1. It doesn't support any ESMTP extensions like STARTTLS or Authentication
2. Others will need to install an smtp server on localhost to run the tests :(

It's a bit tricky this to make ti work for everyone. Maybe I should have a file in the tests directory called "EDIT_THIS_FIRST.php" which defines servers and stuff? But even then I'd need some basic tests to make sure the defined credentials they've given are actually valid.

I'm starting to wonder if I've picked a bad thing to start testing on.

EDIT | Something like:

tests/EDIT_THIS_FIRST.php

Code: Select all

<?php

/*
 For these tests to work you first need to give me some server details.
 */

define('SMTP_SERVER_NAME', 'smtp.host.tld');
define('SMTP_SERVER_PORT', NULL);
define('SMTP_SERVER_USERNAME', NULL);
define('SMTP_SERVER_PASSWORD', NULL);
define('RECEIVER_ADDRESS', 'your@address.com');

?>
I'd then, in each test case do an initial attempt with a basic fsockopen() on the details given, and if that fails it will explain any reason for subsequent tests against swift failing.
User avatar
Maugrim_The_Reaper
DevNet Master
Posts: 2704
Joined: Tue Nov 02, 2004 5:43 am
Location: Ireland

Post by Maugrim_The_Reaper »

Sounds like a plan... If Fakemail is definitely not in the running, then back to what works. Might be a good idea (if errors are noted in the absence of a SMTP resource) to push this test to the bottom of the unit testing pile (let others without the dependency run first so those offline/without the resource are minimally affected).

You can also pass a message through the assert*() methods that is displayed if the assertion fails... I'd make any warning non-fatal.

Reading back what I just wrote, I think this sort of resource testing is going to be difficult in any case. I think separating this out completely from the other tests might be worth while. Have a test.php for non-Connection tests, and a test-connect.php file holding the resource test and other tests which absolutely require it, with your described config file. The requirement for being online is just too much when you could be running these tests frequently... But make it available for final/pre-release testing.

I'm of two minds in other words...:).
Ok now I have a confession. I've duplicated this thread over at SitePoint. Someone there mentioned that this seems more suited to Regression testing since it communicates with the outside world. Would anybody be any the wiser on this?
Sitepoint has highly experienced (if waffling at times) people, their input would be valuable. :)

Did he care to explain Regression Testing? Regression Testing is part and parcel of Unit Testing. If the software "regresses" and a prior problem/bug/issue re-emerges, your unit tests should be capable of uncovering it. Best practice is to create a new test (without fail) to cover any new bugs. This will ensure future test runs deliberately check that bug has not reemerged to haunt you once again. Well, that's one element of regression testing.

JUnit is advertised for example as a regression testing framework! (that's the Java UT lib SimpleTest and co are based on in case you weren't aware).

I figure he's aiming for the separation I suggested - take it out of the equation for a normal test run, and put it in a separate test file for when people are online and can put up with the inevitable latency it will add to tests. I'm heading over to read the Sitepoint thread...
Quick question. I'm just looking at the fakemail stuff. It's written in Perl with a simple dependency right? Is it still OK to include the tests in my tarballs even though people can't run them without an SMTP server running on localhost? Hmm... there are certainly some issues with providing usable tests for something like this
Yes - your test runners are likely developers. You need a LAMP setup (or substitute W for L if that way inclined ;)) to test PHP code, you need a SMTP server or something to fake one, to test your SMTP connection class. Separate it if pushed, but don't exclude it. If peeps can't be bothered to respect your testing than its their problem.

I'm rambling, but to summarise the argument for a separate test for the Connection specifically, and whatever else must (without exception) have a working connection...

1. They require online access
2. They require an external resource and will therefore be slow (most devs using UT will want fast testing since test-runs will be made frequently - for me, it's every 15-30 mins on average depending on what I'm doing, sometimes even more frequently if refactoring code).
3. They require the resource in question - might not be available in all conditions.
Ideally it should be tested against several mail servers since they often behave in different ways. This has always been done before each release in any case so would you say it's reasonable to do the units using fakemail and finish off by doing a more general acceptance test against some real servers before releasing?
That does make sense too. Be sure to include those general acceptance tests for reference.

I think your brick wall is what the library is for - mailing. Even Fakemail is too lite for you test everything.
User avatar
Maugrim_The_Reaper
DevNet Master
Posts: 2704
Joined: Tue Nov 02, 2004 5:43 am
Location: Ireland

Post by Maugrim_The_Reaper »

I'm starting to wonder if I've picked a bad thing to start testing on.
It's the deep end...;). When you move on to another class drop a new reply.
Post Reply