How do handle hard to test features?

Discussion of testing theory and practice, including methodologies (such as TDD, BDD, DDD, Agile, XP) and software - anything to do with testing goes here. (Formerly "The Testing Side of Development")

Moderator: General Moderators

josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

How do handle hard to test features?

Post by josh »

When doing TDD how do you handle hard to test features?

E.g. lazy loading, or testing AJAX vs offline "loading" strategies?

Often the test scenarios are awkward, like deleting and inserting a fake record, and querying to make sure its not found. Although this does and can be encapsulated behind a high level API I'd be interested to see if there are any other "styles" for this type of problem. I guess the central theme here is we are loading something with different strategies (ajax vs offline, lazy vs aggressive, etc..)
User avatar
Jenk
DevNet Master
Posts: 3587
Joined: Mon Sep 19, 2005 6:24 am
Location: London

Re: How do handle hard to test features?

Post by Jenk »

Have you got any examples?

Usually it's just a case of abstracting your API to an extent that has seams, that you can inject mocks/stubs into.

Tests should not use the DB. That's one hard and fast rule I stick to.
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: How do handle hard to test features?

Post by josh »

Jenk wrote:Have you got any examples?
No not really sorry
Usually it's just a case of abstracting your API to an extent that has seams, that you can inject mocks/stubs into.
Yeah, that's a lot easier in java-script because you have event bubbling. What kind of implementation do you use in PHP? Are you usually able to do it without "test code in production"? (eg. its easy enough to add a variable or method for sensing, but isn't that "test code in production"?).
Tests should not use the DB. That's one hard and fast rule I stick to.
I think you mean Unit Tests. I get in this discussion a lot. There are definitely instances where I want (functional) tests to touch the database. Eg. testing a regression in a database adapter class? Personally I write a lot of custom overridden data mapper code I like to have tested. In this case I would even consider the functional test a unit test.. A unit test of the data access object. I guess the only hard and fast rule I would stick to is make sure you can run your tests w/o the database aware tests if needed.

With transaction rollback, there is no test coupling, either.

Thanks for the reply
User avatar
Jenk
DevNet Master
Posts: 3587
Joined: Mon Sep 19, 2005 6:24 am
Location: London

Re: How do handle hard to test features?

Post by Jenk »

Some "test code in production" is not a bad thing. Simply put, if it is making test maintenance/coverage difficult, then it's just making code maintenance difficult.

The most common "test code in production" snippet would be:

Code: Select all

class Foo
{
  pulbic $delegate;
  public function getDelegate()
  {
    if (isset(!$this->delegate)) $this->delegate = new Delegate();
 
    return $this->_delegate;
  }
}
Nothing wrong with that. In production, you'll not even notice the difference. In testing, it's given you a seam to inject your mock/stub.

As far as DB goes.. even with regression testing, I'm not happy having it actually touch the database. The *only* time I am satisfied is with end-to-end tests, which are of course on a sandbox DB which is destroyed after test, and (re)generated at test start.

Wrap all the DB functions in a class, if you aren't using PDO or the like already, with simple methods like:

Code: Select all

class MySQL
{
  public function mysql_query($query, $resource = null)
  {
    return mysql_query($query, $resource);
  }
}
Then instead of calling functions directly, you've got a nice seam to use instead (possibly in conjunction with the above..)

Code: Select all

class Foo
{
  public $db;
 
  public function __construct()
  {
    $this->db = new MySQL();
  }
 
  public function doFoo()
  {
    $this->db->mysql_query("SELECT 'bar' FROM `baz`");
  }
//etc..
}
Then all your test has to do is mock MySQL and inject it before you call upon functionality.
User avatar
Eran
DevNet Master
Posts: 3549
Joined: Fri Jan 18, 2008 12:36 am
Location: Israel, ME

Re: How do handle hard to test features?

Post by Eran »

How do you test methods that invoke complex queries without exercising an actual database? most of the purpose of the test is to check the query is returning correct results, how can you test that with a mock?
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: How do handle hard to test features?

Post by josh »

pytrin wrote:how can you test that with a mock?
He doesn't, he only tests that that function gets called. The query is tested in an "end to end test". For me I put some of this stuff in my normal unit test suite. Since there is transaction rollback there is no state being persisted. It is good to keep them separate so they can be run separately. It increases defect localization at the cost of more (granular) tests to write.

Anyways I guess my problem could be solved with an assertion method called assertRepeatableLaziness( 'setterName', 'methodName', $value)

internally that method would call setterName once with $value, and again with the value of rand(), and then would assert that the getter returned the original value. Is that how you test "laziness"? (E.g. if this class has an instance injected like during testing, it uses that, otherwise it creates a new one)... How do actually write a tests that cover that?

I also have "some test code" in production. They are mostly seams like you have said. I do feel they enhance my design, but at the same time I am conscious of the fact that by the very definition of test code in production its untested code.

Furthermore to try to answer my own question I looked at stuff in Zend that I would find hard to test... what did I find out? They simply skipped writing tests for things like database adapters. Which is nice because it keeps the tests fast but it seems like a pretty crappy decision on their part to make an exception to their "tests required" rule. Is it just general practice to skip database testing? Do you find that you have regressions in that part of the code?
User avatar
Jenk
DevNet Master
Posts: 3587
Joined: Mon Sep 19, 2005 6:24 am
Location: London

Re: How do handle hard to test features?

Post by Jenk »

pytrin wrote:How do you test methods that invoke complex queries without exercising an actual database? most of the purpose of the test is to check the query is returning correct results, how can you test that with a mock?
You test that the query is what is expected. You are also testing the behaviour outcome of the returned results.
User avatar
Eran
DevNet Master
Posts: 3549
Joined: Fri Jan 18, 2008 12:36 am
Location: Israel, ME

Re: How do handle hard to test features?

Post by Eran »

You test that the query is what is expected. You are also testing the behaviour outcome of the returned results.
Not sure what you mean by that. How can you test the query achieves what it sets to achieve without having it parsed by an SQL engine (ie, a database)?
User avatar
Jenk
DevNet Master
Posts: 3587
Joined: Mon Sep 19, 2005 6:24 am
Location: London

Re: How do handle hard to test features?

Post by Jenk »

Because it is not your responsibility to test that the SQL engine parses queries. :)

If you provide XML to a 3rd party XML parser, do you test that it parses XML correctly, or do you just test your app based on the parse results? :)
User avatar
Eran
DevNet Master
Posts: 3549
Joined: Fri Jan 18, 2008 12:36 am
Location: Israel, ME

Re: How do handle hard to test features?

Post by Eran »

Following on your metaphor (which is exactly the same), how do you know you are generating a valid XML? I'm not worried about how the 3rd party service will parse it, I'm just concerned that my XML isn't breaking anywhere. Back to the SQL world - it can get much more complex than generating XML, when you start using aggregate functions, grouping, subqueries and the like. Even a small typo would result in an exception thrown. How can you test that changes to the query don't break anything?
josh
DevNet Master
Posts: 4872
Joined: Wed Feb 11, 2004 3:23 pm
Location: Palm beach, Florida

Re: How do handle hard to test features?

Post by josh »

I'm with pytrin. If you simply assert on the SQL that gets generated you aren't effectively testing anything at all. That would be like testing your PHP code by regexing against it directly.

Personally my rule of thumb is that I follow Data Mapper, not Active Record. Tests for my Models *never* touch a db, but I do test my mappers. So my business logic can be tested without the database present. But I also get good regression testing of the database & the database layer of my app.

In the XML example actually yeah I would want to include the 3rd party system in the harness, at least in a functional test. If that system changes, my app breaks.. therefore I want to know if that happens.
User avatar
Jenk
DevNet Master
Posts: 3587
Joined: Mon Sep 19, 2005 6:24 am
Location: London

Re: How do handle hard to test features?

Post by Jenk »

You'll know it (3rd party tool) is broken when using it.. what is this fear of run-time errors all about? :?

If you load up every tool you use into your test harness/environment, just to test your own apps code, you'll be testing the world soon. What next? Test that Apache handles HTTP transactions reliably? Test that the voltage from your electrical supply is stable? :?

You'll know that the query you are testing against is valid, because you'd have used a query tool (like MySQL Administrator) to help create it yourself first.. there is *no* feasible reason to use a real DB in a test case. It's too slow, it's too complicated (having to set/reset test data) and even transactions cannot be the answer (i.e. what happens when you need to test that your domain can rollback successfully?) plus the overhead of using a transaction.

The art of BDD/TDD is rapid development, with test coverage, not intrinsically test every nook and cranny of every single part of your app, be it your own code, or 3rd party.

There is also the explicit rule of responsibility. If you wrote it, it should be tested. If you didn't, don't test it.
Last edited by Jenk on Wed Feb 10, 2010 7:07 am, edited 1 time in total.
User avatar
Eran
DevNet Master
Posts: 3549
Joined: Fri Jan 18, 2008 12:36 am
Location: Israel, ME

Re: How do handle hard to test features?

Post by Eran »

I really don't understand what you are talking about. Fear of the application breaking at run-time and the risk of regression between changes to source code is exactly what tests are meant to mitigate. Failing queries account for a significant portion of application bugs in any database-reliant application, and not because of database bugs.

The database engine doesn't change between revisions of my code. I'm testing the queries I generate, not the database parsing of it (which I assume is pretty stable). It's pretty rare to stumble upon a database bug for a stable build. The only way to test those queries is to run them through a query parser, ie. a database. I wrote the queries, therefor I should test them as per your logic - and not once on creation as you suggest. I could do that for every other piece of my code, and then I'll be missing on all the advantages of unit testing.
User avatar
Jenk
DevNet Master
Posts: 3587
Joined: Mon Sep 19, 2005 6:24 am
Location: London

Re: How do handle hard to test features?

Post by Jenk »

Why not? create query in a tool and run it, you've then got a *solid* query to test against.. simple example, I've generated a query:

Code: Select all

SELECT `something` FROM `somewhere`
I'd have decided that that is the query I need, before I even write any code. So I need to test that my code will create that query, and that is what is tested.
User avatar
Eran
DevNet Master
Posts: 3549
Joined: Fri Jan 18, 2008 12:36 am
Location: Israel, ME

Re: How do handle hard to test features?

Post by Eran »

This is a very simple static example that you can reasonably claim does not deserve a unit-test (though I don't see the harm in it). However, there are much more complicated use-cases in which it becomes very hard (and tiresome) to test manually all the scenarios for a dynamically generated complicated query. You'd also want to test it with different data (edge-cases) to make sure it doesn't break. All of this can be handled much more easily with unit-tests as opposed to manual testing (which is true for most forms of testing).
Post Reply