Dynamic Test results

Discussion of testing theory and practice, including methodologies (such as TDD, BDD, DDD, Agile, XP) and software - anything to do with testing goes here. (Formerly "The Testing Side of Development")

Moderator: General Moderators

Post Reply
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Dynamic Test results

Post by Chris Corbyn »

I've written a test case which when run makes an unknown number of assertions as the test uses some arbitrary data in order to be a little fairer/well-covered:

Code: Select all

public function testLinesInBodyCannotExceedSpecifiedLength()
	{
		$mime = new MimeExtension();
		$mime->setLineWrap(76);
		$str = "";
		for ($i = 0; $i < 2000; $i++)
		{
			$str .= chr(rand(32, 126)); //Ascii chars
		}
		$mime->setData($str);
		
		$structure = $mime->build();
		$body = substr($structure, strpos($structure, "\r\n\r\n")+4);
		
		$lines = explode("\r\n", $body);
		foreach ($lines as $line)
		{
			$line .= "\r\n"; //lost in explode()
			$this->assertWithinMargin(0, strlen($line), 76);
		}
		
		$mime = new MimeExtension();
		$mime->setLineWrap(1000);
		$mime->setData($str);
		
		$structure = $mime->build();
		$body = substr($structure, strpos($structure, "\r\n\r\n")+4);
		
		$lines = explode("\r\n", $body);
		foreach ($lines as $line)
		{
			$line .= "\r\n"; //lost in explode()
			$this->assertWithinMargin(0, strlen($line), 1000);
		}
		
		$length = rand(10, 700);
		$mime = new MimeExtension();
		$mime->setLineWrap($length);
		$mime->setData($str);
		
		$structure = $mime->build();
		$body = substr($structure, strpos($structure, "\r\n\r\n")+4);
		
		$lines = explode("\r\n", $body);
		foreach ($lines as $line)
		{
			$line .= "\r\n"; //lost in explode()
			$this->assertWithinMargin(0, strlen($line), $length);
		}
	}
Bceuase of the fact it's a bit random, the number of assertions made varies (massively) and thus, the results of the test case always display different numbers. If you keep refreshing the page you just keep seeing a green bar with changing numbers. Would this put people off trusting the test?

Hmm... come to think of it, as simple pattern would probably suffice, although the failure details wouldn't be as clear.
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

Sometimes it's best to supply the sample data in a static file to keep the results consistent.
User avatar
Ambush Commander
DevNet Master
Posts: 3698
Joined: Mon Oct 25, 2004 9:29 pm
Location: New Jersey, US

Post by Ambush Commander »

I agree with Feyd. Furthermore, the randomization does you no good because, in the unlikely case a test case does fail, you have no way of getting the random data!

I generally use input randomization for benchmarking.
User avatar
John Cartwright
Site Admin
Posts: 11470
Joined: Tue Dec 23, 2003 2:10 am
Location: Toronto
Contact:

Post by John Cartwright »

Ambush Commander wrote:I agree with Feyd. Furthermore, the randomization does you no good because, in the unlikely case a test case does fail, you have no way of getting the random data!

I generally use input randomization for benchmarking.
Exactly my sentiments. I would feel horrible if I found my test failed for whatever reason because of random data, because I would be unable to replicate that failure every time (which defeats the purpose of testing).
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

Ok I'll generate ~10 different test samples and just use those. Cheers :) Makes sense I guess.
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

Just to end this thread I actually did this in the end:

Code: Select all

class TestOfMime extends UnitTestCase
{
	protected function makeString($len)
	{
		$ret = "";
		$chr = 32;
		for ($i = 0; $i < $len; $i++) //cycle through the ascii sequence
		{
			$ret .= chr($chr++);
			if ($chr >= 126) $chr = 32;
		}
		return $ret;
	}

// ... snip ....

	public function testLinesInBodyCannotExceedSpecifiedLength()
	{
		$mime = new MimeExtension();
		$mime->setLineWrap(76);
		$mime->setData($this->makeString(2000));
		
		$structure = $mime->build();
		$body = substr($structure, strpos($structure, "\r\n\r\n")+4);
		
		$lines = explode("\r\n", $body);
		foreach ($lines as $line)
		{
			$line .= "\r\n"; //lost in explode()
			$this->assertWithinMargin(0, strlen($line), 76);
		}
		
		$mime = new MimeExtension();
		$mime->setLineWrap(1000);
		$mime->setData($this->makeString(1001));
		
		$structure = $mime->build();
		$body = substr($structure, strpos($structure, "\r\n\r\n")+4);
		
		$lines = explode("\r\n", $body);
		foreach ($lines as $line)
		{
			$line .= "\r\n"; //lost in explode()
			$this->assertWithinMargin(0, strlen($line), 1000);
		}
		
		$mime = new MimeExtension();
		$mime->setLineWrap(999);
		$mime->setData($this->makeString(5003));
		
		$structure = $mime->build();
		$body = substr($structure, strpos($structure, "\r\n\r\n")+4);
		
		$lines = explode("\r\n", $body);
		foreach ($lines as $line)
		{
			$line .= "\r\n"; //lost in explode()
			$this->assertWithinMargin(0, strlen($line), 999);
		}
		
		$mime = new MimeExtension();
		$mime->setLineWrap(999);
		$mime->setData($this->makeString(5003));
		$mime->setEncoding("QP");
		$structure = $mime->build();
		$body = substr($structure, strpos($structure, "\r\n\r\n")+4);
		
		$lines = explode("\r\n", $body);
		foreach ($lines as $line)
		{
			$line .= "\r\n"; //lost in explode()
			$this->assertWithinMargin(0, strlen($line), 999);
		}
		
		$mime = new MimeExtension();
		$mime->setLineWrap(76);
		$mime->setData($this->makeString(4000));
		$mime->setEncoding("base64");
		$structure = $mime->build();
		$body = substr($structure, strpos($structure, "\r\n\r\n")+4);
		
		$lines = explode("\r\n", $body);
		foreach ($lines as $line)
		{
			$line .= "\r\n"; //lost in explode()
			$this->assertWithinMargin(0, strlen($line), 76);
		}
	}
}
lastcraft
Forum Commoner
Posts: 80
Joined: Sat Jul 12, 2003 10:31 pm
Location: London

Post by lastcraft »

Hi.

That's a really long test method. Can you break it into smaller pieces?

yours, Marcus
User avatar
Ambush Commander
DevNet Master
Posts: 3698
Joined: Mon Oct 25, 2004 9:29 pm
Location: New Jersey, US

Post by Ambush Commander »

Uh oh... all my test methods are that size, or longer. :-/
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

lastcraft wrote:Hi.

That's a really long test method. Can you break it into smaller pieces?

yours, Marcus
I should listen to you, being the author and all that stuff :) (thanks!)

But... it's not as if the test does a lot of stuff, it's just long because I wanted to do it with a few different values for a bit of variation. Perhaps the assertions which happen after some encoding has been done can be moved into a new method. Breaking it into smaller methods feels a bit pointless (with the exception of the encoding part) when really, that whole method is just testing that a line-wrap feature is working :)
User avatar
Maugrim_The_Reaper
DevNet Master
Posts: 2704
Joined: Tue Nov 02, 2004 5:43 am
Location: Ireland

Post by Maugrim_The_Reaper »

It seems about right for what it does, Marcus just might mean a short method is easier to digest. Since tests can be documentation, the shorter and more concise an individual test is (even it's only decomposing small functional unit to private methods) the easier it is to understand. It's a fairly common internal refactoring...

In this case you have 4 subunits (calls to new MimeExtension()). Each differs by the encoding used. So four separate tests with four separate methods.
Post Reply