Page 1 of 1

Dynamic Test results

Posted: Thu Nov 23, 2006 4:54 pm
by Chris Corbyn
I've written a test case which when run makes an unknown number of assertions as the test uses some arbitrary data in order to be a little fairer/well-covered:

Code: Select all

public function testLinesInBodyCannotExceedSpecifiedLength()
	{
		$mime = new MimeExtension();
		$mime->setLineWrap(76);
		$str = "";
		for ($i = 0; $i < 2000; $i++)
		{
			$str .= chr(rand(32, 126)); //Ascii chars
		}
		$mime->setData($str);
		
		$structure = $mime->build();
		$body = substr($structure, strpos($structure, "\r\n\r\n")+4);
		
		$lines = explode("\r\n", $body);
		foreach ($lines as $line)
		{
			$line .= "\r\n"; //lost in explode()
			$this->assertWithinMargin(0, strlen($line), 76);
		}
		
		$mime = new MimeExtension();
		$mime->setLineWrap(1000);
		$mime->setData($str);
		
		$structure = $mime->build();
		$body = substr($structure, strpos($structure, "\r\n\r\n")+4);
		
		$lines = explode("\r\n", $body);
		foreach ($lines as $line)
		{
			$line .= "\r\n"; //lost in explode()
			$this->assertWithinMargin(0, strlen($line), 1000);
		}
		
		$length = rand(10, 700);
		$mime = new MimeExtension();
		$mime->setLineWrap($length);
		$mime->setData($str);
		
		$structure = $mime->build();
		$body = substr($structure, strpos($structure, "\r\n\r\n")+4);
		
		$lines = explode("\r\n", $body);
		foreach ($lines as $line)
		{
			$line .= "\r\n"; //lost in explode()
			$this->assertWithinMargin(0, strlen($line), $length);
		}
	}
Bceuase of the fact it's a bit random, the number of assertions made varies (massively) and thus, the results of the test case always display different numbers. If you keep refreshing the page you just keep seeing a green bar with changing numbers. Would this put people off trusting the test?

Hmm... come to think of it, as simple pattern would probably suffice, although the failure details wouldn't be as clear.

Posted: Thu Nov 23, 2006 5:07 pm
by feyd
Sometimes it's best to supply the sample data in a static file to keep the results consistent.

Posted: Thu Nov 23, 2006 8:31 pm
by Ambush Commander
I agree with Feyd. Furthermore, the randomization does you no good because, in the unlikely case a test case does fail, you have no way of getting the random data!

I generally use input randomization for benchmarking.

Posted: Thu Nov 23, 2006 8:47 pm
by John Cartwright
Ambush Commander wrote:I agree with Feyd. Furthermore, the randomization does you no good because, in the unlikely case a test case does fail, you have no way of getting the random data!

I generally use input randomization for benchmarking.
Exactly my sentiments. I would feel horrible if I found my test failed for whatever reason because of random data, because I would be unable to replicate that failure every time (which defeats the purpose of testing).

Posted: Fri Nov 24, 2006 12:45 am
by Chris Corbyn
Ok I'll generate ~10 different test samples and just use those. Cheers :) Makes sense I guess.

Posted: Sat Nov 25, 2006 7:34 am
by Chris Corbyn
Just to end this thread I actually did this in the end:

Code: Select all

class TestOfMime extends UnitTestCase
{
	protected function makeString($len)
	{
		$ret = "";
		$chr = 32;
		for ($i = 0; $i < $len; $i++) //cycle through the ascii sequence
		{
			$ret .= chr($chr++);
			if ($chr >= 126) $chr = 32;
		}
		return $ret;
	}

// ... snip ....

	public function testLinesInBodyCannotExceedSpecifiedLength()
	{
		$mime = new MimeExtension();
		$mime->setLineWrap(76);
		$mime->setData($this->makeString(2000));
		
		$structure = $mime->build();
		$body = substr($structure, strpos($structure, "\r\n\r\n")+4);
		
		$lines = explode("\r\n", $body);
		foreach ($lines as $line)
		{
			$line .= "\r\n"; //lost in explode()
			$this->assertWithinMargin(0, strlen($line), 76);
		}
		
		$mime = new MimeExtension();
		$mime->setLineWrap(1000);
		$mime->setData($this->makeString(1001));
		
		$structure = $mime->build();
		$body = substr($structure, strpos($structure, "\r\n\r\n")+4);
		
		$lines = explode("\r\n", $body);
		foreach ($lines as $line)
		{
			$line .= "\r\n"; //lost in explode()
			$this->assertWithinMargin(0, strlen($line), 1000);
		}
		
		$mime = new MimeExtension();
		$mime->setLineWrap(999);
		$mime->setData($this->makeString(5003));
		
		$structure = $mime->build();
		$body = substr($structure, strpos($structure, "\r\n\r\n")+4);
		
		$lines = explode("\r\n", $body);
		foreach ($lines as $line)
		{
			$line .= "\r\n"; //lost in explode()
			$this->assertWithinMargin(0, strlen($line), 999);
		}
		
		$mime = new MimeExtension();
		$mime->setLineWrap(999);
		$mime->setData($this->makeString(5003));
		$mime->setEncoding("QP");
		$structure = $mime->build();
		$body = substr($structure, strpos($structure, "\r\n\r\n")+4);
		
		$lines = explode("\r\n", $body);
		foreach ($lines as $line)
		{
			$line .= "\r\n"; //lost in explode()
			$this->assertWithinMargin(0, strlen($line), 999);
		}
		
		$mime = new MimeExtension();
		$mime->setLineWrap(76);
		$mime->setData($this->makeString(4000));
		$mime->setEncoding("base64");
		$structure = $mime->build();
		$body = substr($structure, strpos($structure, "\r\n\r\n")+4);
		
		$lines = explode("\r\n", $body);
		foreach ($lines as $line)
		{
			$line .= "\r\n"; //lost in explode()
			$this->assertWithinMargin(0, strlen($line), 76);
		}
	}
}

Posted: Wed Nov 29, 2006 7:07 am
by lastcraft
Hi.

That's a really long test method. Can you break it into smaller pieces?

yours, Marcus

Posted: Wed Nov 29, 2006 4:16 pm
by Ambush Commander
Uh oh... all my test methods are that size, or longer. :-/

Posted: Wed Nov 29, 2006 5:05 pm
by Chris Corbyn
lastcraft wrote:Hi.

That's a really long test method. Can you break it into smaller pieces?

yours, Marcus
I should listen to you, being the author and all that stuff :) (thanks!)

But... it's not as if the test does a lot of stuff, it's just long because I wanted to do it with a few different values for a bit of variation. Perhaps the assertions which happen after some encoding has been done can be moved into a new method. Breaking it into smaller methods feels a bit pointless (with the exception of the encoding part) when really, that whole method is just testing that a line-wrap feature is working :)

Posted: Thu Nov 30, 2006 3:23 am
by Maugrim_The_Reaper
It seems about right for what it does, Marcus just might mean a short method is easier to digest. Since tests can be documentation, the shorter and more concise an individual test is (even it's only decomposing small functional unit to private methods) the easier it is to understand. It's a fairly common internal refactoring...

In this case you have 4 subunits (calls to new MimeExtension()). Each differs by the encoding used. So four separate tests with four separate methods.