Stress testing? If that's the right term for it....

Discussion of testing theory and practice, including methodologies (such as TDD, BDD, DDD, Agile, XP) and software - anything to do with testing goes here. (Formerly "The Testing Side of Development")

Moderator: General Moderators

Post Reply
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Stress testing? If that's the right term for it....

Post by Chris Corbyn »

I'd like to do some strenuous testing of my code, pushing it to boundary case scenarios (i.e. dealing with exceptionally large amounts of data, running an unusually large number of operations in a single request, backtracking repeatedly etc). I was thinking of just bundling this sort of thing into my existing set of tests under SimpleTest, but is there something better I should probably be using?

I'm not really looking to test how well the hardware takes it, but more just to make sure at a logic level the code still does what it's expected to do.

Heh, if I threw it into my existing tests the pass count would be a bit misleading :P
OK
Test cases run: 44/44, Passes: 68478343, Failures: 0, Exceptions: 0
User avatar
Weirdan
Moderator
Posts: 5978
Joined: Mon Nov 03, 2003 6:13 pm
Location: Odessa, Ukraine

Re: Stress testing? If that's the right term for it....

Post by Weirdan »

Actually you can't adequately perform stress tests without any regard to hardware. After all, on a machine with a lot of RAM you might not hit the physical memory limits, on a processor with a lot of 1st lever cache most probably you won't suffer from cache misses, on a processor with short decoding queue you won't suffer that bad from prediction misses due to the conditional logic.

What you can do, though, is to take several different machines / configurations and try to push them to their limit. It will allow you to publish results such as: 'on such and such configuration I was able to send X messages per second, raising memory / processor limits to N allowed to increase this number by a factor of Y', etc.
User avatar
Ambush Commander
DevNet Master
Posts: 3698
Joined: Mon Oct 25, 2004 9:29 pm
Location: New Jersey, US

Re: Stress testing? If that's the right term for it....

Post by Ambush Commander »

It's probably not a good idea to throw these into your regular tests, by virtue of the fact that they will take longer to run.

Also, when things go wrong with large amounts of input, it's usually a limitation of PHP itself, and not your code. For example, PHP's hash implementation can be trivially DOSed because it doesn't use a secure cryptographic function to make sure all the entries don't go in the same bucket. Does anyone care? Not really.

Beyond that, I don't really have any good suggestions.
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Re: Stress testing? If that's the right term for it....

Post by Chris Corbyn »

Thanks guys, both valid points. I guess I'll just leave this stuff out of the public tests and simply do my own ad-hoc testing for peace of mind.
User avatar
Ambush Commander
DevNet Master
Posts: 3698
Joined: Mon Oct 25, 2004 9:29 pm
Location: New Jersey, US

Re: Stress testing? If that's the right term for it....

Post by Ambush Commander »

Then again, there's nothing wrong with making it easy to run the stress-tests; if that means using SimpleTest, do so! Just make sure they don't get in the way of regular testing... nothing worse than a test-suite that takes half an hour to run, so no one runs it.
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Re: Stress testing? If that's the right term for it....

Post by Chris Corbyn »

My test suite is actually really fancy :P It runs tests as completely separate processes (using AJAX or by forking new CLI processes). You can easily have it filter out the stress tests.

Checkout this: https://swiftmailer.svn.sourceforge.net ... simplified

Then open http://localhost/path/to/export/tests in your browser. I haven't implemented the filtering out of "packages" yet but when the groups are expanded you can see them filter as you search.

Typing "header encoder" will show only tests which match "header" AND "encoder".
Typing "header !encoder" will show only tests which match "header" AND NOT "encoder".

You can run them on CLI too if you set the php executable path in tests/config.php.

"php tests/run.php" to run all tests.
"php tests/run.php Name_Of_TestClass" to run a single test
"php tests/run.php /pattern/" to run tests matching the regex

I did think about releasing it if I polished it off a bit. I could probably throw in same flag to "include stress tests", or just put $this->skipIf(!CONFIG_RUN_STRESS) in the test cases.

I guess they're not really stress tests as such... just end-to-end tests for boundary cases.
User avatar
Maugrim_The_Reaper
DevNet Master
Posts: 2704
Joined: Tue Nov 02, 2004 5:43 am
Location: Ireland

Re: Stress testing? If that's the right term for it....

Post by Maugrim_The_Reaper »

Just curious if you intend using ab with Apache? Probably the thing to use for some kinds of stress testing. Putting the tests under SimpleTest isn't a bad idea if they're segregated somehow. Gives a framework to organise each test, and then ab or your custom stress runner is just a few calls away.
alex.barylski
DevNet Evangelist
Posts: 6267
Joined: Tue Dec 21, 2004 5:00 pm
Location: Winnipeg

Re: Stress testing? If that's the right term for it....

Post by alex.barylski »

What are you trying to stress? Logic, obviously should be tested using unit tests. As you are already aware, stress testing isn't reaqlly about testing that, it's more testsing how your code performs under stressful situations, like when an operation exceeds memory limits or deals with some other environmental issue, such as a downed database. I think of stress testing as a way to assist you in picking out hotspots in your code - in need of tuning.

Because PHP handles many critical errors for you, like memory overload, there is little you can do, except maybe monitor memory consumption and stop before the script haults.

Then there is testing how well your code performs under distributed environments, which is where I personally think PHP stress testing should focus on. How well does your application scale? Does it begin bogging down or even breaking when concurrent requests are made?

Your code should remain thread safe unless you implement some kind of IPC via SHMOP or similar to communicate between processes.

How do you stress test to emulate a distrubuted environment? Under CLI at least you could just have a shell script invoke a bunch of processes and see what happens I guess. On my home network, I have a firefox extension on several computers and I program then to basically begin firing away at the server making requests, uploading large data, etc. Doesn't emulate a real world environment very well (5 computers making simultaneous requests) but I suppose it gives me some idea.

In short, I'm not sure stress testing a PHP application is really worth while, as the environment typically handles most of that for you. You never have to worry about race conditions or deadlock, again, unless you use IPC...which most stress tests are trying to discover.

How does your application handle under great stress or multiple loads in a distrubuted environment:
1) When memory peaks - the script aborts (not much you can do here but buy new hardware and increase the default allowed)
2) When two or more processes are spawned, they will likely never collide. You might run into a situation when you use a file for caching data and forget to check for locks and/or lock it period.

Anyways, I would basically writeup a simple shell script that created the object(s) in question and blasted them with megabytes of quasi-data. I would then have another script invoke this said script several hundred times a minute to emulate a heavy load on a web site, all from the command line. :P
User avatar
Chris Corbyn
Breakbeat Nuttzer
Posts: 13098
Joined: Wed Mar 24, 2004 7:57 am
Location: Melbourne, Australia

Re: Stress testing? If that's the right term for it....

Post by Chris Corbyn »

Hockey wrote:What are you trying to stress? Logic, obviously should be tested using unit tests. As you are already aware, stress testing isn't reaqlly about testing that, it's more testsing how your code performs under stressful situations, like when an operation exceeds memory limits or deals with some other environmental issue, such as a downed database. I think of stress testing as a way to assist you in picking out hotspots in your code - in need of tuning.
This is exactly what I was getting at. I wanted to put my code through some really tough situations and hope I code identify some weak points which I could then improve.
Because PHP handles many critical errors for you, like memory overload, there is little you can do, except maybe monitor memory consumption and stop before the script haults.
Because this is for SwiftMailer (v4) I was wanted to test more things like network flooding and excessive disk access (if file caching is enabled). Also, things like the ability to attach massive files (100MB+) without exhausting memory limits.
Post Reply