-
Bug
-
Resolution: Fixed
-
Medium
-
2.0.2
-
None
-
Linux duke 2.6.22-14-generic #1 SMP Sun Oct 14 23:05:12 GMT 2007 i686 GNU/Linux
Clover reports a "failure" when running Instinct specifications that expect an exception. Instinct does not propagate the exception (perhaps JUnit 4 does), and instead handles it during specification running.
Here's an example of what causes the failure, a screenshot is attached shoing how this manifests in the HTML report).
@Specification(expectedException = IllegalArgumentException.class, withMessage = "Population cannot be chunked with chunk size (-1) less than 1") public void shouldNotAcceptNegativeChunkSize() { new PopulationToChunkMapperImpl(-1); }
Changing the code to the following "works" (i.e. doesn't cause Clover to fail).
public void shouldNotAcceptNegativeChunkSize() { try { new PopulationToChunkMapperImpl(-1); throw new RuntimeException("Expected exception not thrown"); } catch (IllegalArgumentException e) { // all good } }
Clover is enabled/configured as follows:
<clover-setup initstring="${coverage.db}" tmpdir="${coverage.results.dir}/tmp" flushpolicy="threaded" flushinterval="${coverage.flush.interval}"> <fileset refid="main.covered.fileset"/> <testsources dir="${spec.src.dir}"> <testclass> <testmethod annotation="Specification"/> </testclass> </testsources> </clover-setup>
Maybe it's got something to do with how clover is configured. For example Instinct will also run spec methods (analogous to JUnit test methods) beginning with "should" and "must" however these do not appear to work correctly in all circumstances.
This may be related to this issue: http://jira.atlassian.com/browse/CLOV-90.
- is related to
-
-
- Closed
-
[CLOV-94] Clover incorrectly reports failing Instinct specifications when they expect an exception
Clover now understands how Instinct specifies expected exception types and reports correctly on these. We still need to do more work to make sure Clover also tests the expected exception message if specified. http://jira.atlassian.com/browse/CLOV-154
We are planning to support better built-in test result collection in the future.
For now (as of 2.0.3) however - Clover will be able to read a set of TEST-*.xml files as produced by the Ant Junit task, or the Maven Surefire plugin.
Tom Adams
added a comment - Hey Nick,
Thanks again for the quick comments, you guys take this stuff seriously don't you! Good to see things are still looking good after your move to Atlassian. And well done for supporting Aussie open source also!
I thought about the issue a bit more and realised that you must be doing something like wrapping the method body in order to detect exceptions. As you suggest, Instinct will reflectively run spec methods and if they're annotated to expect failures, will allow the expected exception to mean a successful run.
Even though it's a de facto standard, I still think the XML way is a hack. There are "test" frameworks that support other formats like agiledox.
Is it not possible to specify another way to figure out if test failures are expected? Basically you need to figure out if the test framework was expecting the error. TestNG, JUnit and Instinct all support the annotation way of documenting expected exceptions. It'd be pretty easy to keep this stuff around so that you can reconcile it. A cleaner (but more work) would be to write custom (or allow users or framework developers) checkers for each test framework. You could hand each one the java.lang.reflect.Method that failed and the exception you caught and allow the checker to confirm it's expected or not.
That's probably getting a bit carried away though 
It's great to know we can include the "failed" test coverage in the results, for our project it increases the coverage by 12%. This will suit us nicely.
Thanks,
Tom
Tom Adams
added a comment - Hey Nick,
Thanks again for the quick comments, you guys take this stuff seriously don't you! Good to see things are still looking good after your move to Atlassian. And well done for supporting Aussie open source also!
I thought about the issue a bit more and realised that you must be doing something like wrapping the method body in order to detect exceptions. As you suggest, Instinct will reflectively run spec methods and if they're annotated to expect failures, will allow the expected exception to mean a successful run.
Even though it's a de facto standard, I still think the XML way is a hack. There are "test" frameworks that support other formats like agiledox.
Is it not possible to specify another way to figure out if test failures are expected? Basically you need to figure out if the test framework was expecting the error. TestNG, JUnit and Instinct all support the annotation way of documenting expected exceptions. It'd be pretty easy to keep this stuff around so that you can reconcile it. A cleaner (but more work) would be to write custom (or allow users or framework developers) checkers for each test framework. You could hand each one the java.lang.reflect.Method that failed and the exception you caught and allow the checker to confirm it's expected or not.
That's probably getting a bit carried away though It's great to know we can include the "failed" test coverage in the results, for our project it increases the coverage by 12%. This will suit us nicely.
Thanks,
Tom G'Day Tom,
The beta version of Clover2 used the XML test result files. The format used by the Ant JUnit task seems to have been made a quasi standard for test result xml. It is produced by the Maven-Surefire test runner and TestNG also. As part of the Clover2 integration process - users had to ensure that their JUnit task was configured with an xml formatter. We will be re-enabling this feature in the next release.
Having Clover2 display the test results is quite useful. Like you say - it is a single place to look. By default - Clover2 will exclude coverage attributed to a test which failed. This can be turned off by setting the includeFailedTestCoverage attribute on the <current/> element to 'true'.
I've just raised this JIRA to allow test results to be turned off as you suggest. I'll hopefully get you a build of 2.0.3 next week.
One thing I'm still not sure of is why clover thinks it's a failure. There's no XML to parse and Instinct does not propagate the exception.
How does Instinct detect a test pass? If your test method throws any exception, Clover will mark this test as failed. It does no parsing or processing of annotations. Clover is catching the Exception, marking the test as failed, and then throwing the exception again. In other words - Clover "sees" the exception before instinct does.
Cheers,
Nick
Tom Adams
added a comment - Hey Nick,
No, Instinct does not generate any XML report files (it may do so in the future). This approach sounds like a bit of a hack to me, as a lot of projects don't even run JUnit with XML reports turned on. As clover is hooked into the build, it may be better to use another mechanism to detect build failures. One way I can think of off the top of my head is to monitor ant warning level logging. Another approach would be to define custom regex's to run over ant output, for example "FAILED" would be a way to detect Instinct failures, JUnit would be similar.
Another approach would be to allow this feature to be turned off. It may be a silly question, but why does clover need to detect failures? Surely this is the job of the tool (JUnit, Instinct, TestNG, etc.) to generate reports, etc. I think it's nice to have this feature (it means I don't have to write a tool for Instinct to generate XML), but maybe not everyone wants it.
One thing I'm still not sure of is why clover thinks it's a failure. There's no XML to parse and Instinct does not propagate the exception. The only thing I can think of is that it's parsing the source code and seeing the "expectedException" attribute or the Exception.class.
Thanks,
Tom
Tom Adams
added a comment - Hey Nick,
No, Instinct does not generate any XML report files (it may do so in the future). This approach sounds like a bit of a hack to me, as a lot of projects don't even run JUnit with XML reports turned on. As clover is hooked into the build, it may be better to use another mechanism to detect build failures. One way I can think of off the top of my head is to monitor ant warning level logging. Another approach would be to define custom regex's to run over ant output, for example "FAILED" would be a way to detect Instinct failures, JUnit would be similar.
Another approach would be to allow this feature to be turned off. It may be a silly question, but why does clover need to detect failures? Surely this is the job of the tool (JUnit, Instinct, TestNG, etc.) to generate reports, etc. I think it's nice to have this feature (it means I don't have to write a tool for Instinct to generate XML), but maybe not everyone wants it.
One thing I'm still not sure of is why clover thinks it's a failure. There's no XML to parse and Instinct does not propagate the exception. The only thing I can think of is that it's parsing the source code and seeing the "expectedException" attribute or the Exception.class.
Thanks,
Tom Hi Tom,
Thank you for reporting this. It is directly related to CLOV-90. Clover does not recognize any Test annotations such as @Specification.
The solution we are planning will involve parsing your test result TEST-*.xml files. This is infact how Clover2 originally got your test results. Initially this will be a fallback option only for tests which use Annotations to determine passes/fails. We may try to get a little smarter in the future with annotations but this will not be a top priority since reading TEST-*.xml files will also work.
Do you generate xml results for your Inspect tests? Is the format the same as that produced by the Ant JUnit task?
Cheers,
Nick
I have documented Instinct and all other supported frameworks here: Clover+test+detection