High The various filtering options (statement contexts, method contexts, exclusions, etc) do not seem to be applied consistently across the various clover2 maven goals: clover2:clover, clover2:check: clover2:log. The discrepancies between clover2:clover and clover2:check essentially render clover2:check useless as it cannot be relied upon for accurate threshold reporting.
The attached maven project demonstrates the following use cases:
1. A method covered by tests
2. A method not covered by tests
3. A property method excluded via the property method context
4. A constructor method excluded via the constructor method context
5. A method filtered using a custom method context
6. A statement filtered using a custom statement context
7. A file filtered using the "exclusion" configuration option
In order to execute the example, unzip the attached zip file and run the following:
mvn clean clover2:setup test clover2:aggregate clover2:clover clover2:check clover2:log
Things to note:
Clover2:clover generates correctly. Coverage is at 50% and 71.4% of the code is correctly filtered out.
Clover2:check reports coverage at 66.7%. Now in this case clover2:check is reporting coverage higher than clover2:clover. I'm not sure how it's getting there, but in my production examples the opposite tends to be the case: clover2:check reports lower coverage than clover2:clover.
Clover2:log reports coverage at 33.3%:
[INFO] Clover Coverage Report
Coverage Timestamp: Fri Mar 02 12:26:09 EST 2012
Coverage Overview -
Coverage:-
Methods: 2/7 (28.6%)
Statements: 4/12 (33.3%)
Branches: 1/2 (50%)
Total: 33.3%
Complexity:-
Avg Method: 1.1428572
Density: 0.6666667
Total: 8
If I change my maven execution to the following:
mvn clean clover2:instrument test clover2:aggregate clover2:clover clover2:check clover2:log
where clover2:setup is swapped out for clover2:instrument, the coverage check matches the report: 50%.
As you can see, none of these tools seems to be reporting the same thing. You might say "Leo, why not use clover2:instrument so at least clover2:clover and clover:2check" match? Well, there's a problem with clover2:instrument as well: http://forums.atlassian.com/thread.jspa?threadID=22387
I've been using clover for some time now (~5+ years). Your main competitive advantage over the other guys (jacoco, cobertura, emma, etc) is your advanced filtering and reporting capabilities. The frustrating thing is that in these 5 years none of these features have ever seemed to work consistently across maven goals nor in integration with other tools like STS and Sonar.
Please make it a priority to ensure that these features are enabled consistently and tested for as part of every release. You know, ensure you have proper code coverage on your code coverage product. 
Please feel free to integrate my example project into your automated regression suite!
Thanks!
1.
|
Update Confluence documentation |
|
Closed | Marek Parfianowicz |
|
[CLOV-1064] Lack of Filtering/Reporting Consistency Across Maven Goals
Leo Hart
added a comment - You can go ahead and close the bug. If my application is still having a problem I'll open a new jira. It seems like the controlled use cases are all passing.
Thanks!
Leo Hart
added a comment - You can go ahead and close the bug. If my application is still having a problem I'll open a new jira. It seems like the controlled use cases are all passing.
Thanks! Hi Leo, do you have any comments to this bug? I'd like to close it, otherwise.
Thanks for information. As a matter of fact, I'm surprised that there are still some differences. 
I would propose checking how your application/test code is split into different folders (src/main, src/test) and/or named (JUnit3 naming, JUnit4 annotations etc) . Things which were fixed by us (me & Michael) were mainly related with filtering of test classes (so they are not included in statistics) and detection of test code.
Leo Hart
added a comment - The example project looks good: all the percentages match up.
I've tried running this on my app and I'm getting different percentages. Going to try to run that again (it takes a while).
Leo Hart
added a comment - The example project looks good: all the percentages match up.
I've tried running this on my app and I'm getting different percentages. Going to try to run that again (it takes a while). Hi Leo, please have a look at our latest build attached in clover-3.1.6-snapshot-v2.zip+clover-3.1.6-snapshot-v2.z01
Thanks Leo. That was a regression caused by me. We will get you a new build asap.
Leo Hart
added a comment - I think you can disregard the "Access is Denied" issue for now. I was able to duplicate it using 3.1.4, but only by using that combination of Maven goals, which I don't normally use all at once.
The "Failed to render syntax highlights for..." error does seem to be introduced with 3.1.6, however. Let me know if there's more info I can provide to help debug that.
Leo Hart
added a comment - I think you can disregard the "Access is Denied" issue for now. I was able to duplicate it using 3.1.4, but only by using that combination of Maven goals, which I don't normally use all at once.
The "Failed to render syntax highlights for..." error does seem to be introduced with 3.1.6, however. Let me know if there's more info I can provide to help debug that. Leo Hart
added a comment - I was able to test this on our development project and ran across some interesting results. For one, there seems to be some issue introduced with this release as my clover execution no longer completes. I get a "Access Denied" message on a file during packaging.
I also am getting lots of these messages as part of the clover2:clover goal:
[ERROR] Failed to render syntax highlights for SomeJavaClass.java
java.lang.NullPointerExceptionjava.lang.NullPointerException
at clover.org.codehaus.groovy.antlr.parser.GroovyLexer.require(GroovyLexer.java:309)
at clover.org.codehaus.groovy.antlr.parser.GroovyLexer.mSTRING_CTOR_END(GroovyLexer.java:2321)
at clover.org.codehaus.groovy.antlr.parser.GroovyLexer.mSTRING_LITERAL(GroovyLexer.java:1985)
at clover.org.codehaus.groovy.antlr.parser.GroovyLexer.nextToken(GroovyLexer.java:468)
at clover.org.codehaus.groovy.antlr.parser.GroovyLexer$1.nextToken(GroovyLexer.java:258)
at com.atlassian.clover.reporters.html.source.groovy.GroovySourceTraverser.traverse(GroovySourceTraverser.java:178)
at com.atlassian.clover.reporters.html.source.groovy.GroovySourceRenderer.render(GroovySourceRenderer.java:18)
at com.cenqua.clover.reporters.html.source.SourceRenderHelper.render(SourceRenderHelper.java:311)
at com.cenqua.clover.reporters.html.source.SourceRenderHelper.gatherSrcRenderInfo(SourceRenderHelper.java:117)
at com.cenqua.clover.reporters.html.source.SourceRenderHelper.insertLineInfosForFile(SourceRenderHelper.java:47)
at com.cenqua.clover.reporters.html.RenderFileAction.insertLineInfos(RenderFileAction.java:284)
at com.cenqua.clover.reporters.html.RenderFileAction.render(RenderFileAction.java:131)
at com.cenqua.clover.reporters.html.RenderFileAction.call(RenderFileAction.java:115)
at com.cenqua.clover.util.CloverExecutors$LoggingCallable.call(CloverExecutors.java:86)
at clover.retrotranslator.edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:178)
at clover.retrotranslator.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1061)
at clover.retrotranslator.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
at java.lang.Thread.run(Thread.java:662)
I'm going to keep digging into those.
Going through this made me realize that my test case doesn't demonstrate aggregation. I've refactored it to include 2 modules, module-a and module-b. With that change it seems like the numbers are a tad off in the parent project (project attached).
Execute this against the parent project:
mvn clean clover2:instrument test clover2:aggregate clover2:clover clover2:check clover2:log
I've attached the complete log, but you'll notice that module-a correctly reports:
- check: 50%
- log: 50%
- clover: 50%
module-b correctly reports:
- check: 33.3%
- log: 33.3%
- clover: 33.3%
The parent project, clover-example seems to incorrectly report:
- check: 40% <--correct
- log: 57.1% <--incorrect
- clover: 40% <--correct
So that's something you might want to look at.
I'll let you know if I learn more about those other errors; let me know if there's some more info I can provide.
Leo Hart
added a comment - I was able to test this on our development project and ran across some interesting results. For one, there seems to be some issue introduced with this release as my clover execution no longer completes. I get a "Access Denied" message on a file during packaging.
I also am getting lots of these messages as part of the clover2:clover goal:
[ERROR] Failed to render syntax highlights for SomeJavaClass.java
java.lang.NullPointerExceptionjava.lang.NullPointerException
at clover.org.codehaus.groovy.antlr.parser.GroovyLexer.require(GroovyLexer.java:309)
at clover.org.codehaus.groovy.antlr.parser.GroovyLexer.mSTRING_CTOR_END(GroovyLexer.java:2321)
at clover.org.codehaus.groovy.antlr.parser.GroovyLexer.mSTRING_LITERAL(GroovyLexer.java:1985)
at clover.org.codehaus.groovy.antlr.parser.GroovyLexer.nextToken(GroovyLexer.java:468)
at clover.org.codehaus.groovy.antlr.parser.GroovyLexer$1.nextToken(GroovyLexer.java:258)
at com.atlassian.clover.reporters.html.source.groovy.GroovySourceTraverser.traverse(GroovySourceTraverser.java:178)
at com.atlassian.clover.reporters.html.source.groovy.GroovySourceRenderer.render(GroovySourceRenderer.java:18)
at com.cenqua.clover.reporters.html.source.SourceRenderHelper.render(SourceRenderHelper.java:311)
at com.cenqua.clover.reporters.html.source.SourceRenderHelper.gatherSrcRenderInfo(SourceRenderHelper.java:117)
at com.cenqua.clover.reporters.html.source.SourceRenderHelper.insertLineInfosForFile(SourceRenderHelper.java:47)
at com.cenqua.clover.reporters.html.RenderFileAction.insertLineInfos(RenderFileAction.java:284)
at com.cenqua.clover.reporters.html.RenderFileAction.render(RenderFileAction.java:131)
at com.cenqua.clover.reporters.html.RenderFileAction.call(RenderFileAction.java:115)
at com.cenqua.clover.util.CloverExecutors$LoggingCallable.call(CloverExecutors.java:86)
at clover.retrotranslator.edu.emory.mathcs.backport.java.util.concurrent.FutureTask.run(FutureTask.java:178)
at clover.retrotranslator.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1061)
at clover.retrotranslator.edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:575)
at java.lang.Thread.run(Thread.java:662)
I'm going to keep digging into those.
Going through this made me realize that my test case doesn't demonstrate aggregation. I've refactored it to include 2 modules, module-a and module-b. With that change it seems like the numbers are a tad off in the parent project (project attached).
Execute this against the parent project:
mvn clean clover2:instrument test clover2:aggregate clover2:clover clover2:check clover2:log
I've attached the complete log, but you'll notice that module-a correctly reports:
check: 50%
log: 50%
clover: 50%
module-b correctly reports:
check: 33.3%
log: 33.3%
clover: 33.3%
The parent project, clover-example seems to incorrectly report:
check: 40% <--correct
log: 57.1% <--incorrect
clover: 40% <--correct
So that's something you might want to look at.
I'll let you know if I learn more about those other errors; let me know if there's some more info I can provide. Leo Hart
added a comment - Marek, I tested it out on the project I sent you and everything looks good. Coverage is reported as 50% across the project. I'm going to try to run it on our development project as an extra measure and see if the results are the same, but so far so good! Thanks!
Leo Hart
added a comment - Marek, I tested it out on the project I sent you and everything looks good. Coverage is reported as 50% across the project. I'm going to try to run it on our development project as an extra measure and see if the results are the same, but so far so good! Thanks! 
Thanks to you, Leo, for providing the controlled use case. We really appreciate it.