update: Both of these features have been rolled into a JUnit functional testing extension called JFunc available from sourceforge.net
I'm currently using Junit to do integration testing (or functional testing), rather than unit testing. This involves a number of differences from what the standard Junit methodology would suggest. It's important to understand that the modifications I've made aren't necessarily applicable to everyone who's doing unit testing, but for those of you out there who feel that sometimes Junit gets in the way of doing other sorts of testing this may help.
For instance, a lot of the information my tests require isn't available until the test is run or is generated during runtime and needs to be passed between the tests. Fixtures are not really applicable to this sort of testing. Another thing that began to happen was I started mangling my own test method names in a sort of ad hoc way of passing arguments to the tests. To try and fix these two situations, I've made the following extensions to Junit.
For me this makes the feel of Junit much more natural. It also provides compile time checks that wouldn't normally happen until runtime. This is made possible by Proxies for classes.
These two pieces of code should illustrate the difference. Old Style:
TestSuite suite = new TestSuite();
suite.add(new TestTest("testPassed"));
suite.add(new TestTest("testFailed"));
suite.add(new TestTest("testErrored"));
// can't do this with the current TestCase in Junit
// suite.add(new TestTest("testArgs", new Object[] { "ugly" }));
return suite;
New Style:
ProxyTestSuite suite = new ProxyTestSuite();
//suite.oneTestInstancePerTest(false);
TestTest tc = (TestTest) suite.getTestProxy(new TestTest());
tc.testPassed();
tc.testFailed();
tc.testErrored();
// TestProxies can accept arguments
// tc.testArgs("pretty");
return suite;
These two pieces of code are effectively the same (neglecting the commented out testlet that accepts arguments), except one is cleaner, easier to read in my opinion, and has more checks that happen at compile time. Also the new style allows for one to optionally choose between one test instance per test (default) or one test instance for all tests of that type.
The source is available here. It includes a shell script to make and run the stuff. (Would somebody from the Windows world send me a batch file.)
I will be posting a link to this site in response to some messages being passed around in Junit mailing list. I appreciate any direct feedback, but posting to the Junit mailing list would probably be a better place if you're interested in a lively discussion.
-shane (at) terraspring.com
update: Both of these features have been rolled into a JUnit functional testing extension called JFunc available from sourceforge.net
old update: In response to a request for the option to change the assert behavior, between the standard "fatal" assertions, and the new non-standard "non-fatal" assertions, I created a new patch. Fatal assertions imply only one failure per test, whereas the non-fatal assertions allow for multiple failures per test. This behavior can be manipulated via an accessor on the Assert class (see example below). By default it is initialized to the standard fatal assertions.
There has been a lot of discussion on the JUnit mailing list about allowing multiple failures in a test. Some people insist that with the right amount of refactoring there should be no need to introduce this flexibility. I, however, found this wasn't the case (see above as my testing might significantly differ from yours). This restriction hampered my testing, as I had to reflect more upon the framework than my tests themselves. Being forced to choose between having only one assert per test which provided more information, rather than the more logical aggregations of asserts which provided less information.
The change isn't radical by any means. It simply allows your tests to have multiple failures attributed to them. Which means if you're say comparing a collection of values, you get a report back of all the elements that didn't match, not just the first one that caused an AssertionFailureError to be thrown.
I've put together a patch against the 3.7 sources that you can apply. Or you can simply use this junit3.7b.jar (source is included in the jar). Update: Old patch and jar file: junit3.7a.patch, junit3.7a.jar.
The test I added to SimpleTest looks like this:
public SimpleTest(String name) {
super(name);
setFatal(false); // use non-fatal asserts
}
public void testMultipleFailures() {
fail("failing once");
fail("failing twice");
fail("failing once again");
}
Running test runner presents the results as follows (stack traces have been removed for sake of brevity):
$ java -classpath junit3.7a.jar junit.textui.TestRunner junit.samples.SimpleTest .F.E.FF.FFF Time: 0.034 There was 1 error: 1) testDivideByZero(junit.samples.SimpleTest)java.lang.ArithmeticException: / by zero There were 6 failures: 1) testAdd(junit.samples.SimpleTest)junit.framework.AssertionFailedError 2) testEquals(junit.samples.SimpleTest)junit.framework.AssertionFailedError: Size expected:<12> but was:<13> 3) testEquals(junit.samples.SimpleTest)junit.framework.AssertionFailedError: Capacity expected:<12.0> but was:<11.99> 4) testMultipleFailures(junit.samples.SimpleTest)junit.framework.AssertionFailedError: failing once 5) testMultipleFailures(junit.samples.SimpleTest)junit.framework.AssertionFailedError: failing twice 6) testMultipleFailures(junit.samples.SimpleTest)junit.framework.AssertionFailedError: failing once again FAILURES!!! Tests run: 4, Failures: 6, Errors: 1
The Swing and AWT TestRunner should work fine as well. The only inconsistency that this has introduced is the listing of failures may exceed the number of failed tests displayed on the status line, which is slightly confusing since they've typically had a one to one correspondence until now.
Update: I left TestResult.java alone in the latest patch. In the prior patch I augmented it slightly to report the number of unique tests that had failed, that way the status line "Tests run: x, Failures: y, Errors: z" would still retain the familiar x = y + z, but decided to do away with that.
-shane (at) terraspring.com