Tinderbox for PHP
Posted: Sat Nov 25, 2006 9:21 pm
Mozilla has a neat little tool called Tinderbox. It performs live data about the broken-ness of builds by compiling and testing the source code under a variety of different environments.
While PHP is not as volatile and does not compiling, and since the project I'm thinking about has only one developer on it, much of this functionality is not necessary. But there is one thing that is extremely appealing: automated testing of a library/application under multiple environments and differing versions of PHP.
There's a number of problems that have to be overcome first, of course.
1. Instead of Tinderbox's fairly small list of machines performing the builds, this system would have several OS's combined with dozens of versions of PHP. How would this data be reported in a logical fashion? (Listing only broken versions) How would you choose to drop a version of PHP, i.e. ignore broken tests on that configuration? (Push some XML configuration file) How would you get all the versions to co-exist together and be callable in a portable fashion? (I know how to do this for Win32, not so much for Unix)
2. What would be used to test the application? (Simpletest) How would this data be collected in an easily-parsable format for aggregation into the entire build status report? (XML)
3. How would the client machines report to the central machine the status of the test? (Tinderbox uses email). How would you efficiently make the client machine also the master machine, for one-machine builds? (email self, access XML test results over local system)
4. How would client machines know when to perform a test? (do with cron or have post-commit hook that notifies client machines when there is a checkin)
5. How would the master machine interface with the repository (SVN/CVS/etc) in order to determine who broke the application, output detailed revision info about the tests? How would the client machines know which version of the code their using? (svnlook) (note, there are no SVN bindings PHP library implemented yet)
Does this sound like an interesting project?
While PHP is not as volatile and does not compiling, and since the project I'm thinking about has only one developer on it, much of this functionality is not necessary. But there is one thing that is extremely appealing: automated testing of a library/application under multiple environments and differing versions of PHP.
There's a number of problems that have to be overcome first, of course.
1. Instead of Tinderbox's fairly small list of machines performing the builds, this system would have several OS's combined with dozens of versions of PHP. How would this data be reported in a logical fashion? (Listing only broken versions) How would you choose to drop a version of PHP, i.e. ignore broken tests on that configuration? (Push some XML configuration file) How would you get all the versions to co-exist together and be callable in a portable fashion? (I know how to do this for Win32, not so much for Unix)
2. What would be used to test the application? (Simpletest) How would this data be collected in an easily-parsable format for aggregation into the entire build status report? (XML)
3. How would the client machines report to the central machine the status of the test? (Tinderbox uses email). How would you efficiently make the client machine also the master machine, for one-machine builds? (email self, access XML test results over local system)
4. How would client machines know when to perform a test? (do with cron or have post-commit hook that notifies client machines when there is a checkin)
5. How would the master machine interface with the repository (SVN/CVS/etc) in order to determine who broke the application, output detailed revision info about the tests? How would the client machines know which version of the code their using? (svnlook) (note, there are no SVN bindings PHP library implemented yet)
Does this sound like an interesting project?