How much should one test?

Discussion of testing theory and practice, including methodologies (such as TDD, BDD, DDD, Agile, XP) and software - anything to do with testing goes here. (Formerly "The Testing Side of Development")

Moderator: General Moderators

Post Reply
Ree
Forum Regular
Posts: 592
Joined: Fri Jun 10, 2005 1:43 am
Location: LT

How much should one test?

Post by Ree »

I am trying SimpleTest right now (after following the nice introductory tutorial on lastcarft.com) and I picked a simple (I thought so) class to write - MySQL DB class. I haven't coded much at all and a question came up - how extensive the tests should be? Should I test ALL possible variations of method usage? Here are my tests:

Code: Select all

define('SIMPLE_TEST', '../simpletest/');

require_once(SIMPLE_TEST . 'unit_tester.php');
require_once(SIMPLE_TEST . 'reporter.php');

require_once('../classes/class.Database.php');

//Existing (correct) DB info
define('db_host', 'localhost');
define('db_name', 'techno_db');
define('db_user', 'root');
define('db_pass', '');

class TestOfDatabase extends UnitTestCase
{
  function TestOfDatabase()
  {
    $this->UnitTestCase('Test of Database');
  }

  function testSuccessfullConnection()
  {
    $db = &new Database(db_host, db_name, db_user, db_pass);
    $this->assertIdentical($db->getError(), '');
  }

  function testUnsuccessfullConnection()
  {
    $db = &new Database('foo', db_name, db_user, db_pass);
    $this->assertIdentical($db->getError(), 'UnableToConnect');
  }

  function testSuccessfullConnectionButWrongDB()
  {
    $db = &new Database(db_host, 'foo', db_user, db_pass);
    $this->assertIdentical($db->getError(), 'NoDatabase');
  }

  function testDisconnectAfterSuccessfullConnection()
  {
    $db = &new Database(db_host, db_name, db_user, db_pass);
    $db->disconnect();
    $this->assertIdentical($db->getError(), '');
  }

  function testDisconnectAfterUnsuccessfullConnection()
  {
    $db = &new Database('foobar', db_name, db_user, db_pass);
    $db->disconnect();
    $this->assertIdentical($db->getError(), 'UnableToDisconnect');
  }
}
    
$test = &new TestOfDatabase();
$test->run(new HtmlReporter());
The class itself so far:

Code: Select all

class Database
{
  var $connection;
  var $error;

  function Database($host, $name, $user, $password)
  {
    $this->connection = @mysql_connect($host, $user, $password, true);
    if ($this->connection)
    {
      $select = mysql_select_db($name, $this->connection);
      if ($select)
      {
        $this->error = '';
      } else
      {
        $this->error = 'NoDatabase';
      }
    } else
    {
      $this->error = 'UnableToConnect';
    }
  }

  function disconnect()
  {
    if (!$this->connection)
    {
      $this->error = 'UnableToDisconnect';
      return;
    }
    mysql_close($this->connection);
  }

  function getError()
  {
    return $this->error;
  }
}
You can see the tests are trivial and extremely simple... I imagine all possible variations when running a query (query() method), which would need quite some new tests:

Code: Select all

function testSuccessfullConnectionGoodQuery()
  {
  }

  function testSuccessfullConnectionBadQuery()
  {
  }

  function testUnsuccessfullConnectionAndQuery()
  {
  }

  function testSuccessfullConnectionWrongDBGoodQuery()
  {
  }

  function testSuccessfullConnectionWrongDBBadQuery()
  {
  }

/* ... */
Do I have to write all these variations? For example, I test a case when a db object is created and disconnect() method is run immediately. This type of usage will never happen since it's dumb, but still, let's leave it there. Now how do I know that disconnect() will behave correctly when used together with query() method and that I will get the correct and not overwritten error message (if there is any) at the end? That's another test with db object creation, and running query() and disconnect() methods. You can think of many variations. Should one test all?

For example, McGruff does not test the disconnect() method in his MySQL class at all and seems fine with it. viewtopic.php?t=36162&start=15

So I am not exactly sure about testing all these variations... maybe some tests are redundant?

Well, would like to find out general guidelines on what tests should be picked.
McGruff
DevNet Master
Posts: 2893
Joined: Thu Jan 30, 2003 8:26 pm
Location: Glasgow, Scotland

Re: How much should one test?

Post by McGruff »

You're absolutely right about testing disconnect(). The class has changed a little since then but I still hadn't caught that :oops:

You've got some good, descriptive names which make the test case easy to read. Tests will be used as documentation and its important (as always) to write clear code.
Ree wrote:Should I test ALL possible variations of method usage?
I think you're on the right track: unit tests should be quite fine-grained. They should describe in detail how the class ought to behave so that, when something isn't working, you can pinpoint the problem quickly. With a database class, that means doing pretty much what you've done: testing with successful/failed connections, good & bad queries, error returns in different error states and so on.

At the same time, be careful not to test more than you have to. Unecessary constraints will make it harder to find solutions and might interfere with refactoring. Ultimately, it's up to you to decide exactly what is and isn't relevant. Some things will choose themselves and others will be more a matter of what kind of behaviour you prefer.

Another thing to watch out for is to try not to think about the code while you're writing the tests. It can be an easy trap to fall into. The test logic shouldn't any make assumptions which are only valid with a specific implementation. This will work as "quality control" for a specific solution where the assumptions are valid but it's no good for refactoring where you must have implementation-agnostic tests.

Remember you don't have to get everything 100% right first time (I never do, as you can see :) ). With the tests in place you've at least got some solid ground on which to build.
Ree
Forum Regular
Posts: 592
Joined: Fri Jun 10, 2005 1:43 am
Location: LT

Re: How much should one test?

Post by Ree »

McGruff wrote:At the same time, be careful not to test more than you have to.
It's the single thing I am most worried about. The very purpose of the whole thing is testing the behaviour 'from all sides', so normally one would want to write many tests. But how do I know that I am not writing any extra/redundant tests?
User avatar
Nathaniel
Forum Contributor
Posts: 396
Joined: Wed Aug 31, 2005 5:58 pm
Location: Arkansas, USA

Re: How much should one test?

Post by Nathaniel »

Ree wrote:
McGruff wrote:But how do I know that I am not writing any extra/redundant tests?
I would say experience.
Roja
Tutorials Group
Posts: 2692
Joined: Sun Jan 04, 2004 10:30 pm

Re: How much should one test?

Post by Roja »

Ree wrote:But how do I know that I am not writing any extra/redundant tests?
The answer "experience" isn't a wrong answer, but it may be incomplete. What you learn during that experience is some criteria that help you come to that conclusion.

One could be if you have a test that in (dozens? hundreds?) of revisions of the program never changes, or tests a codepath that never changes.

Ideally, tests should help drive your design. If you write a test, and then write code that fulfills that test, and that code never has to change, there is little value in repeating the test going forward.

However, (and this is where experience helps), knowing which parts *will* change in the future can be extremely hard to predict, and many will tell you (correctly) that eliminating a test just because you dont think the code will change is also wrong.

Its a judgement call, but unchanging code is a pretty reasonable point to add to the list of things that you don't need to continue testing once it is passed. (In my opinion).
McGruff
DevNet Master
Posts: 2893
Joined: Thu Jan 30, 2003 8:26 pm
Location: Glasgow, Scotland

Re: How much should one test?

Post by McGruff »

Roja wrote:Its a judgement call, but unchanging code is a pretty reasonable point to add to the list of things that you don't need to continue testing once it is passed. (In my opinion).
You should really always test everything except maybe for trivial methods like simple getters and setters (even then in php4 you might want to be ultra cautious about making sure an object reference is returned rather than a copy which could produce some hard to figure out problems down the line).

Unchanging code might pass all its tests now but fail when you try to install on a different platform later. Some filesystem functions don't work the same with symlinks as they do with windows shortcuts. A particular SERVER var might not be available, etc etc. Or, a change to class A might cause class B to fail elsewhere in the app. You have to treat it as a whole.

Then there's refactoring. The thing is, you can never be sure a class isn't going to change. The whole point of XP is to be "agile" and fluid and you need a complete test suite to guide that.

Or you, you might simply want to add new tests as new issues come to light: maybe to check a validation rule can cope with a XSS attack.

Usually a full set of tests would be run regularly while you're developing, again while you test an update on a staging server, and again when you do the final install.
deltawing
Forum Commoner
Posts: 46
Joined: Tue Jun 14, 2005 2:55 pm

Post by deltawing »

Just as an aside, I've been testing a lot in Java lately, and I tend to have a testGettersAndSetters() method. This saves on multiple methods for something which is really trivial. Admittedly though, I'll write the getters and setters and then write my unit test, but the test just makes sure I haven't done anything stupid. It might test for default values, call the setters, then test for set values.
Ree
Forum Regular
Posts: 592
Joined: Fri Jun 10, 2005 1:43 am
Location: LT

Post by Ree »

So I wrote the tests for the DB connection class and wrote the class itself. Here's how it looks like (test names should be self-explanatory):

Code: Select all

<?php

define('SIMPLE_TEST', '../simpletest/');

require_once(SIMPLE_TEST . 'unit_tester.php');
require_once(SIMPLE_TEST . 'reporter.php');

require_once('../classes/class.MySQLDatabase.php');

define('db_host', 'localhost');
define('db_name', 'test');
define('db_user', 'root');
define('db_pass', '');

class TestOfMySQLDatabase extends UnitTestCase
{
  var $connection;

  function TestOfMySQLDatabase()
  {
    $this->UnitTestCase('Test of MySQL Database');
  }

  function setUp()
  {
    $this->connection = mysql_connect(db_host, db_user, db_pass, true);
    $this->query("CREATE DATABASE " . db_name);
    mysql_select_db(db_name, $this->connection);
    $this->query("CREATE TABLE test_table (test_field TEXT)");
    $this->query("INSERT INTO test_table VALUES ('test_value')");
    $this->db = &new MySQLDatabase(db_host, db_name, db_user, db_pass);
  }

  function tearDown()
  {
    mysql_query("DROP DATABASE " . db_name, $this->connection);
    mysql_close($this->connection);
  }

  function testSuccessfullConnection()
  {
    $this->assertIdentical($this->db->error(), false);
    $this->assertIdentical($this->db->getError(), '');
  }

  function testGoodQuery()
  {
    $result = $this->db->query("SELECT test_field FROM test_table");
    $this->assertTrue(is_object($result));
    $this->assertIdentical($result->fetch(), array('test_field' => 'test_value'));
    $this->assertIdentical($this->db->error(), false);
    $this->assertIdentical($this->db->getError(), '');
  }

  function testBadQuery()
  {
    $result = $this->db->query("foo");
    $this->assertIdentical($this->db->error(), true);
    $this->assertIdentical($this->db->getError(), 'SQLError');
  }

  function testGoodManipulate()
  {
    $this->db->manipulate("INSERT INTO test_table VALUES ('another_value')");
    $this->assertIdentical($this->db->error(), false);
    $this->assertIdentical($this->db->getError(), '');
    $result = $this->db->query("SELECT test_field FROM test_table WHERE test_field='another_value'");
    $this->assertTrue(is_object($result));
    $this->assertIdentical($result->fetch(), array('test_field' => 'another_value'));
    $this->assertIdentical($this->db->error(), false);
    $this->assertIdentical($this->db->getError(), '');
  }

  function testBadManipulate()
  {
    $this->db->manipulate("foo");
    $this->assertIdentical($this->db->error(), true);
    $this->assertIdentical($this->db->getError(), 'SQLError');
  }

  function testErrorsAreClearedAfterEachQuery()
  {
    $this->db->manipulate("foo");
    $this->assertIdentical($this->db->error(), true);
    $this->assertIdentical($this->db->getError(), 'SQLError');
    $result = $this->db->query("SELECT test_field FROM test_table");
    $this->assertIdentical($this->db->error(), false);
    $this->assertIdentical($this->db->getError(), '');
    $result = $this->db->query("foo");
    $this->assertIdentical($this->db->error(), true);
    $this->assertIdentical($this->db->getError(), 'SQLError');
    $this->db->manipulate("INSERT INTO test_table VALUES ('booh')");
    $this->assertIdentical($this->db->error(), false);
    $this->assertIdentical($this->db->getError(), '');
  }

  function testSuccessfullConnectionDisconnect()
  {
    $this->db->disconnect();
    $this->assertIdentical($this->db->error(), false);
    $this->assertIdentical($this->db->getError(), '');
  }

  //Connection problem tests (separate db object is created)
  function testSuccessfullConnectionWrongDB()
  {
    $db = &new MySQLDatabase(db_host, 'booh', db_user, db_pass);
    $this->assertIdentical($db->error(), true);
    $this->assertIdentical($db->getError(), 'NoDB');
    $result = $db->query("SELECT test_field FROM test_table");
    $this->assertIdentical($db->error(), true);
    $this->assertIdentical($db->getError(), 'NoDB');
    $db->manipulate("INSERT INTO test_table VALUES ('booh')");
    $this->assertIdentical($db->error(), true);
    $this->assertIdentical($db->getError(), 'NoDB');
  }

  function testUnsuccessfullConnection()
  {
    $db = &new MySQLDatabase('booh', db_name, db_user, db_pass);
    $this->assertIdentical($db->error(), true);
    $this->assertIdentical($db->getError(), 'UnableToConnect');
    $result = $db->query("SELECT test_field FROM test_table");
    $this->assertIdentical($db->error(), true);
    $this->assertIdentical($db->getError(), 'UnableToConnect');
    $db->manipulate("INSERT INTO test_table VALUES ('booh')");
    $this->assertIdentical($db->error(), true);
    $this->assertIdentical($db->getError(), 'UnableToConnect');
  }

  function testSuccessfullConnectionWrongDBDisconnect()
  {
    $db = &new MySQLDatabase(db_host, 'booh', db_user, db_pass);
    $db->disconnect();
    $this->assertIdentical($db->error(), false);
    $this->assertIdentical($db->getError(), '');
  }

  function testUnsuccessfullConnectionDisconnect()
  {
    $db = &new MySQLDatabase('booh', db_name, db_user, db_pass);
    $db->disconnect();
    $this->assertIdentical($db->error(), true);
    $this->assertIdentical($db->getError(), 'UnableToDisconnect');
  }

  function query($sql)
  {
    mysql_query($sql, $this->connection);
  }
}
    
$test = &new TestOfMySQLDatabase();
$test->run(new HtmlReporter());

?>
And this is the class:

Code: Select all

<?php

require_once('class.MySQLResultIterator.php');

class MySQLDatabase
{
  var $connection;
  var $select;
  var $error;

  function MySQLDatabase($host, $name, $username, $password)
  {
    $this->connection = @mysql_connect($host, $username, $password, true);
    if ($this->connection)
    {
      $this->select = mysql_select_db($name, $this->connection);
      if ($this->select)
      {
        $this->error = '';
        return;      
      }
      $this->error = 'NoDB';
      return;
    }
    $this->error = 'UnableToConnect';
  }

  function query($sql)
  {
    if ($this->select)
    {
      $result = mysql_query($sql, $this->connection);
      if (!$result)
      {
        $this->error = 'SQLError';
        return;
      }
      $this->error = '';
      return $iterator = &new MySQLResultIterator($result);
    }
  }

  function manipulate($sql)
  {
    if ($this->select)
    {
      $result = mysql_query($sql, $this->connection);
      if (!$result)
      {
        $this->error = 'SQLError';
        return;
      }
      $this->error = '';
    }
  }

  function disconnect()
  {
    if ($this->connection)
    {
      $this->error = '';
      mysql_close($this->connection);
      return;
    }
    $this->error = 'UnableToDisconnect';
  }

  function getError()
  {
    return $this->error;
  }

  function error()
  {
    return (bool)$this->error;
  }
}

?>
When I did all this, now I wonder again: aren't I doing too much testing? There's quite a lot of code in the tests... Do you think all the tests are proper or are some of them unneccessary at all? You can easily notice, that tests pretty much cover all possible variations of method usage. Now since I pretty much finished it, any comments on this would be greatly appreciated.
lastcraft
Forum Commoner
Posts: 80
Joined: Sat Jul 12, 2003 10:31 pm
Location: London

Post by lastcraft »

Hi...
Ree wrote: When I did all this, now I wonder again: aren't I doing too much testing? There's quite a lot of code in the tests... Do you think all the tests are proper or are some of them unneccessary at all? You can easily notice, that tests pretty much cover all possible variations of method usage. Now since I pretty much finished it, any comments on this would be greatly appreciated.
On average I tend to write about as much test code as I do production code, and that seems to be pretty typical. That's an average though, and some classes are crucial to the operation of the system. Stuff that interacts with the outside world, also attracts more tests. I used to call these boundary classes, but Fowler calls then gateways, which I must admit is a better name. Your connection is definitely a gateway.

I think your testing for this class is excellent. No one's an expert in this relatively new field, though.

Because gateway code is a pain to test, once I have my knarly class tested, I tend to mock it everywhere else. Once you start mocking, test code drops dramatically as you move into the "tell don't ask" world of OO.

I would say test until you are confident. That's why testing getters and setters is not worthwhile, it's usually obvious that they will work.

yours, Marcus
User avatar
Ambush Commander
DevNet Master
Posts: 3698
Joined: Mon Oct 25, 2004 9:29 pm
Location: New Jersey, US

Post by Ambush Commander »

That's why testing getters and setters is not worthwhile, it's usually obvious that they will work.
Unless, of course, they have extra side-effects. I'd say test when it's complicated or non-intuitive.
lastcraft
Forum Commoner
Posts: 80
Joined: Sat Jul 12, 2003 10:31 pm
Location: London

Post by lastcraft »

Hi.

Yes, I am clearly talking about simple getters and setters here.

yours, Marcus
Post Reply