Page 1 of 2

BDD/TDD for Applications

Posted: Wed May 07, 2008 6:19 pm
by Ollie Saunders
So how many of you actually use TDD or BDD development for application code? It seems to me that if I need to write a library or even just a couple of class helpers TDD and BDD are great but as soon as you are using a database and writing application code things become really tricky and the effort and expertise involved is no longer worth it.

Firstly there are a lot of technical reasons why testing becomes a lot of work: you need two databases and you have to examine HTML output.

You can try and break down HTML generation into testable checks by using component based generation but in my experience those components don't offer significant flexibility and so they have to be written each time and then you have a situation where you have to write 3x the amount of code than you would have with an untested template.

Should I be looking into browser-based testing systems?

Re: BDD/TDD for Applications

Posted: Wed May 07, 2008 6:32 pm
by Chris Corbyn
I do. I don't examine HTML output though since that's a really fragile test scenario. You can't really test a GUI in a non-fragile manner so that comes down to manual testing. Even the automated user interface testing tools are apparently fragile.

I test my controllers and my model. That's more or less like testing a library really. As long as you have Request and Response objects you can pass to your controllers then you can test for redirects and 404 status codes etc. I often have to "log in" a mock user with incorrect permissions and then test that the controller throws an AccessException and all these kinds of things.

Code: Select all

function testGuestUserHasAccessDenied() {
  $this->_setActiveUser($this->_guestUser);
  try {
    $this->_controller->executeAction('someAction');
    $this->fail('Access should be denied t guest users');
  } catch (AccessException $e) {
  }
}
When testing that the controller changes the model in the correct way you simply put the model in a certain state during set-up, run the controller, then check the state of the model is what you expected the controller to change it too. Basically most of the testing you do is business-oriented, not UI oriented :) You need a decent architecture to manage it though (dependency lookup of model components, separate DB with test data etc).

It's handy to have your schema in a .sql file so that the setUp() of each test can rebuild a fresh schema in a test database.

Re: BDD/TDD for Applications

Posted: Thu May 08, 2008 3:49 am
by Jenk
I don't see why you "need" two databases.. unless you are referring to a database installed on localhost, and a database on the live machine, which you will have regardless of use of T/BDD?

I use it all the time, everywhere, for everything. I can't think of any scenario where T/BDD can't be employed.

Re: BDD/TDD for Applications

Posted: Thu May 08, 2008 5:40 am
by Chris Corbyn
One database serves as a platform for testing only.... it gets wiped and set up afresh before each test. The other is the one you use to manually interact with the system and to demo the app with. It's a lot easier to get a feel for how the app works if it's not constantly being wiped when you run your system tests. Not wiping the database (or at least the relevant parts of it) before running your system tests is dangerous though... it can lead to a stale fixture situation very easily.

Re: BDD/TDD for Applications

Posted: Thu May 08, 2008 7:32 am
by Jenk
Tests should not hit the database.. your datalayer will have tests that it has run elsewhere, previously and is a "released" product.

Re: BDD/TDD for Applications

Posted: Thu May 08, 2008 7:50 am
by Chris Corbyn
Jenk wrote:Tests should not hit the database.. your datalayer will have tests that it has run elsewhere, previously and is a "released" product.
In an ideal world tests wouldn't hit the database no. But in reality you write tests which do use a real database... you just try minimize the number of them you write because they run slower. Take for example your domain object layer. You write your finders/mappers which are looking for data across multiple tables using complex SQL. You do write tests for those. More often than not it's easier just to write tests for your controllers by keeping theses DOCs in tact rather than mocked. Trying to mock everything in your domain just makes tests harder to understand (and probably more fragile).

I'd be happy to concede and change my opinions if I could see some real-world tests for a database driven system which don't use a database during their testing. In fact, it would be amazing to see it done well without jumping through hoops :)

Re: BDD/TDD for Applications

Posted: Thu May 08, 2008 11:23 am
by Jenk
It depends on the API of your datalayer (and/or mapper should you be fortunate enough to use a mapper.) My only example is GLORP for Smalltalk, which has very intensive tests (and over 300 of them,) which allows me to completely avert using the actual database for my domain tests.

If your datalayer allows you to construct statements without raw SQL writing, then mocking would be my preffered choice for tests, even if it does mean writing 100's of lines for the mocks. This may not be the case for you, of course. Infact I'd probably still mock even if it were raw SQL. Progressive development is key to my working life, so it wouldn't seem that much of a task for me. (/end gloating)

Re: BDD/TDD for Applications

Posted: Fri May 09, 2008 3:19 am
by Maugrim_The_Reaper
I'm with JCart - I avoid using a real database as normal practice. All my data layer libraries at the end of the day are mockable and I can simulate returns form any specific database call. Usually I just maintain stock test data is a loose format like XML or plain text.

To each his own however ;).

Re: BDD/TDD for Applications

Posted: Fri May 09, 2008 10:14 am
by Ollie Saunders
I mean this might be a subjective point but personally I feel that if you can use the real thing then that's going to be better than a mock especially if it's less effort to set up.

Re: BDD/TDD for Applications

Posted: Sun May 11, 2008 9:25 pm
by Jenk
A couple of points to consider with that.

Your tests are not concise - you are testing more than just the domain logic, you are also testing the specific data accessor, the connectivity of webapp to db, the data structure, etc. etc. Tests/Specs should be concise.

You are now completely dependant on a particular type of DB - usually not a problem, but you could also be limiting your self to a particular version which can be problematic. Tied in with the above, your tests which are primarily for domain logic, will fail for data accessing problems - again this goes back to the point about tests/specs being concise.

Re: BDD/TDD for Applications

Posted: Sun May 11, 2008 9:39 pm
by Chris Corbyn
I'd agree with you. It's just that I've not successfully experienced replacing my database layer with test doubles entirely before. You surely still have at least *some* end-to-end tests which access the database? I mean, tests for those components which are a direct interface for the DB such as ORM classes. Even if you're using something like Propel which is tested; you add your own domain logic to the peer classes it produces so you should be testing that logic.

I'll have to play around more with replacing my data-accessing components with test doubles since there's probably a bit of an art to doing it in a flexible manner.

Re: BDD/TDD for Applications

Posted: Sun May 11, 2008 9:49 pm
by Jenk
Cost vs Reward will obviously play a massive role here, but perhaps this is also an indication of a potential design flaw?

You'll have a data accessor, i.e. the object that retrieves directly from the DB and has an API that allows you to complete any DB task without the need to send it SQL. You'll have your ORM objects, that map from your domain object to the data layer. I don't see where the need for direct access is for, other than the tests of the data accessor - which is unavoidable - but this data accessor would be in your library/framework that will/should have tests independant of those for your application?

To go back to GLORP on Smalltalk, I use PostgreSQL with it, and it translates Smalltalk to SQL for me - for anyone who can read Smalltalk, this is an example "query" :

Code: Select all

myObject := glorpSession
  readOneOf: MyObject
  where: [ :eachObject | eachObject anAccessor asLowercase = 'foo' ]

Re: BDD/TDD for Applications

Posted: Mon May 12, 2008 3:37 am
by Maugrim_The_Reaper
You surely still have at least *some* end-to-end tests which access the database?
Of course. I used to run acceptance tests on real applications eventually which requires a complete duplicate of the production system including database with a data population. I'm not sure if that's what you mean though.

The way I do things, there are numerous testing strategies and each has a particular strength. The strength of unit testing/specs is code design. Design cares little for data unless it is code specifically connecting to a database because it literally calls mysqli_connect() or similar ;). Other than that, data is mocked to ensure the class being tested, which does not directly use a database, is isolated. Testing to a database breaks that isolation.

Here's a question I sometimes raise for this. You're building an RSS parser. Do you a) connect to localhost to consume RSS from a feed you setup for the purpose, or b) jam RSS from a text file into the parser. Most answer b. Now replace RSS with DB Data.

The other mention is that mocking data is time consuming. Depends on how you do it. If population of data is a repetitive, common task that varies little than I pull out my toolbox and extend the unit testing framework.

Finally - as Jenk said. Presumably you've already testing, or are relying on tests covering, the database driver/interaction class. Right there you're breaking test isolation and adding an inter-test dependency (chained failures).

There is one really good counter-argument though. It really could be far easier for some projects ;). I have seen a situation where faking data was not pretty at all and we all opted to pull from a pre-constructed database instead. It was easier, and we accepted the costs of doing so. We even finished on time...

Re: BDD/TDD for Applications

Posted: Tue May 13, 2008 11:43 am
by McGruff
A connection object encapsulates a database connection but not the messages sent via the connection. If I want to order a pizza all the way from Italy a telephone is just the start: I'll also need to know how to speak Italian. Similarly, data access classes need to know how to compose messages in the server's own language, and these messages will have to be verified.

As a rule, at an application boundary I think it's better to test with a real resource. The behaviour of an external resource, like a pizza server, is not under my control and so I can't mock it reliably.

With a mocked connection, tests will define sql string expectations:

Code: Select all

 
$connection->expectOnce('query', 'select foo from bar')
 
However, a simple string comparison is far too sensitive. It will fail with minor variations of upper/lower case, different whitespace, and so on, all of which are ignored by the database server. I'd be over-specifying the problem, excluding many viable solutions. You might be prepared to live with that for very simple queries but with more complex sql it could quickly become annoying. You might refactor a piece of code then find that AND clauses are written in a different order: exactly the same query but now there's a failing test, a false negative, to deal with.

With a mocked connection, you'd really need an SqlExpectation class (SimpleTest has a nice feature which allows expectation objects to be substituted for parameter expectations). However, this would have to duplicate every last detail of parsing logic in the target dbms - an impossible task. You could almost guarantee it would never be wholly bug free.

Luckily there's a foolproof query parser all ready to go: the dbms. I think it's best to use a real connection object, a real database, and real data all set up and torn down for each test. Unless the test suites are very large, most modern PCs can handle all the creating and dropping easily without getting out of breath.
Jenk wrote:Your tests are not concise - you are testing more than just the domain logic, you are also testing the specific data accessor, the connectivity of webapp to db, the data structure, etc. etc.
That can be a good thing. You could have some sql which in itself is a perfectly valid query but nevertheless doesn't work with the target database. Maybe a tinytext field is truncating data. Maybe the foo table which you specified doesn't exist at all. That won't show up with a mock.

Although there is more work to do creating the fixtures, I don't think tests have to be any less concise. The fixture set up can be tucked away in methods or classes of its own, as necessary. No ping can (and probably always should) be set as a skip condition.
ole wrote:Should I be looking into browser-based testing systems?
Check out the SimpleTest web tester.

Re: BDD/TDD for Applications

Posted: Tue Jul 15, 2008 6:35 am
by alex.barylski
How do you determine if the SQL statement is valid if you don't test the actual result? Assuming that SQL is used as an interface to a storage medium?

If it were something else, like a file operation, how do you determine whether the file permissions are correct if you don't test the actual result?

I enjoyed BDD at first but after much consideration, although it allows easier testing than TDD...something doesn't feel right about it.