Group tests - memory exhaustion

Discussion of testing theory and practice, including methodologies (such as TDD, BDD, DDD, Agile, XP) and software - anything to do with testing goes here. (Formerly "The Testing Side of Development")

Moderator: General Moderators

Post Reply
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Group tests - memory exhaustion

Post by Chris Corbyn »

I've got this huge group test which runs 33 testcases in itself. It's using about 10MB of memory and thus, not running on the default memlimit. Anyone else ever get this problem? It does loads of regex and stuff like that and also creates quite a lot of mock objects which of course uses eval().

I think I'm going to have to categorise my tests into smaller groups.
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

I think smaller groups could help. 33 cases seems quite large for a group. :P
User avatar
Ambush Commander
DevNet Master
Posts: 3698
Joined: Mon Oct 25, 2004 9:29 pm
Location: New Jersey, US

Post by Ambush Commander »

Hmm... the thing is, after the test-cases finish, the memory should be freed up by PHP's garbage collector. Are you doing anything funky in that respect?

I noticed, recently, that my scripts started hitting the memory limit, and I had to bump up the PHP INI limit. I couldn't tie it to anything: it just happened. Puzzled too, although it's not too annoying.
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

No, it's all the mock objects I'm fairly sure. I split my group tests into 4 smaller ones and it's all hunky dory.
User avatar
Maugrim_The_Reaper
DevNet Master
Posts: 2704
Joined: Tue Nov 02, 2004 5:43 am
Location: Ireland

Post by Maugrim_The_Reaper »

Does the default memory limit really matter when testing? I just configure it up to 32MB for huge test runs and leave it there unless testing is actually of memory usage. Granted for the average user it's a good idea to have smaller groups so they don't need to hunt for their ini file, but having a testall option isn't bad or evil ;).
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

I agree. I was going to do this, but on further thought I decided that if people were going to download it - most on 8MB limits - it would look better if they didn't even have to think about changing their setup to run the tests. I had somebody contact me to say they were unsure about why the tests would not run before I started doing my own digging. I just envisaged getting a lot of support emails if I left the runAllTests.php file in there :P
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

Wow, I so totally did not think that would be settable at runtime :)
User avatar
feyd
Neighborhood Spidermoddy
Posts: 31559
Joined: Mon Mar 29, 2004 3:24 pm
Location: Bothell, Washington, USA

Post by feyd »

d11wtq wrote:Wow, I so totally did not think that would be settable at runtime :)
You wouldn't really think so, but sadly it is. I don't know if it will be in PHP 6, but in 4 and 5, it certainly is. :)
lastcraft
Forum Commoner
Posts: 80
Joined: Sat Jul 12, 2003 10:31 pm
Location: London

Post by lastcraft »

Hi.

At Wordtracker we have the memory limit set to 256MB for testing :(.

Due to bugs in various versions of PHP 4 and 5, the mocks have to find a global link to the test case. This confuses the garbage collector and keeps both old test cases and old mock alive it seems. Certainly memory is not being given back despite attempts at liberal sprinklings of unset().

You can run test cases in their own test cases, but it's currently a bit clumsy. In an upcoming version I'll be fixing this usability problem, and possibly using the sandbox feature of PHP 5.2 for TestSuites.

I don't find the memory limit too much of a problem in practice though.

yours, Marcus
User avatar
AKA Panama Jack
Forum Regular
Posts: 878
Joined: Mon Nov 14, 2005 4:21 pm

Post by AKA Panama Jack »

feyd wrote:What about ini_set('memory_limit', XXX)?
You have to compile PHP to support that if you are using any version of PHP before 5.2.1. Most hosting companies will prevent you from changing the memory limit as well.

http://us3.php.net/manual/en/ini.core.p ... mory-limit

The other thing to remember is that PHP will not deallocate memory when you unset an object. There isn't a way to clear memory when an object is finished being used. You can unset the object variable but all that does is remove the pointers to the object and doesn't destroy the object in memory.

So you need to break your test into smaller groups of objects if you want it to run in the 8meg limit. If this is a test that will be supplied to others you should do this because you can't rely on hosts allowing clients to change their memory limits via PHP.

Maybe with PHP 6 they will actually allow you to destroy an object so the memory is deallocated but right now that's just not possible.
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Post by Chris Corbyn »

lastcraft wrote:Hi.

At Wordtracker we have the memory limit set to 256MB for testing :(.

Due to bugs in various versions of PHP 4 and 5, the mocks have to find a global link to the test case. This confuses the garbage collector and keeps both old test cases and old mock alive it seems. Certainly memory is not being given back despite attempts at liberal sprinklings of unset().

You can run test cases in their own test cases, but it's currently a bit clumsy. In an upcoming version I'll be fixing this usability problem, and possibly using the sandbox feature of PHP 5.2 for TestSuites.

I don't find the memory limit too much of a problem in practice though.

yours, Marcus
It's not a problem really :) These are tests; you can hardly cast judgements on an application by the fact it's unit tests are very intensive and use a lot of memory. If the tests were only there for my own usage in TDD I'd have simply upped my memory limit but I'm happy splitting the tests up because I know other people have tried running the AllTests file on shared hosts and been put off by it. It works, I'm happy ;)
Post Reply