CLMVN-144 The :log and :check goals seem to use different sums, or calculate them differently ?

Moved from: https://studio.plugins.atlassian.com/browse/CLMVN-143 and CLMVN-144
=====================================

The :log and :check goals seem to use different sums, or calculate tehm differently ?

It seems that the :log and :check goals use different figures ? http://pastie.org/2654554

It seems that the :log and :check goals use different figures, and calculate thresholds differently ? See the following output:

[INFO] [clover2:instrument

{execution: instrument-and-check}

]
[INFO] [clover2:log

{execution: instrument-and-check}

]
[INFO] Clover Version 3.1.0, built on May 31 2011 (build-821)
[INFO] Loaded from: /Users/gjoseph/.m2/repository/com/cenqua/clover/clover/3.1.0/clover-3.1.0.jar
[INFO] Clover: Open Source License registered to Magnolia CMS.
[INFO] Clover Coverage Report
Coverage Timestamp: Fri Oct 07 12:35:22 CEST 2011

Coverage Overview -
Coverage:-
Methods: 105/142 (73.9%)
Statements: 399/525 (76%)
Branches: 92/176 (52.3%)
Total: 70.7%
Complexity:-
Avg Method: 1.7535211
Density: 0.47428572
Total: 249

[INFO] [clover2:check

{execution: instrument-and-check}

]
[INFO] Checking for coverage of [70] for database [/Users/gjoseph/Dev/magnolia/svn/community/magnolia/magnolia-templating/target/clover/clover.db]
[INFO] Clover Version 3.1.0, built on May 31 2011 (build-821)
[INFO] Loaded from: /Users/gjoseph/.m2/repository/com/cenqua/clover/clover/3.1.0/clover-3.1.0.jar
[INFO] Clover: Open Source License registered to Magnolia CMS.
[INFO] Coverage check FAILED
[ERROR] The following coverage targets for null were not met:
Total coverage of 66% did not meet target of 70%
[ERROR] Total coverage of 66% did not meet target of 70%

=========

Nick Pellow added a comment - 09/Oct/11 4:19 PM

Hi Gregory,

This is a known limitation with Clover. The Log task includes the coverage of your test classes as well, whereas the Check task will not.
You can check this by excluding all your test classes from instrumentation.

Cheers,
Nick

=========

Grégory Joseph added a comment - 10/Oct/11 2:29 AM

Hi Nick,

Thanks for the heads up. That also explains why the difference is bigger when we have more tests

This is perhaps not the best place for discussing this (let me know), but what are the use cases for instrumenting test classes ? When would I be interested - in checking how much my tests are covered... by other tests ? (I can see it would be useful when writing a test framework, but other than that ?)

=========

Grégory Joseph added a comment - 14/Oct/11 3:54 AM - edited

Nick,

<includesTestSourceRoots>false</includesTestSourceRoots> seems to do the trick indeed, but as per the doc, it disables the reporting of per-test coverage.

The reason we're using the log goal in the first place, is because the check goal does not output the current coverage when it passes. It does output a few things, but then just mentions "passed" - while it does output the current coverage when it fails. As far as I'm concerned, if it would just output the current coverage, I'd just ditch the call to log. I'd suggest this:

Current messages:

Coverage check PASSED
Total coverage of 34.3% did not meet target of 50%

Proposed messages:

Total coverage of 34.3% successfully met target of 50%
Total coverage of 34.3% did not meet target of 50%

=========

Grégory Joseph added a comment - 03/Nov/11 9:59 AM

Another discrepancy I've noticed, is that the :log goal doesn't use the contextFilter. That's also quite a burden, since we'd like our dev to look at the log, and raise the bar (i.e the targetThreshold) if they see the coverage has increased. WDYT ?

=========

Nick Pellow added a comment - 09/Nov/11 7:29 PM

Hi Gregory,

I like your idea of improving the Clover logging in the Clover Check task.
I have created https://jira.atlassian.com/browse/CLOV-1036 for this, however can't give you an exact release date just yet.
I could potentially get you a development build containing a fix for the improved logging in clover:check though. Let me know if you are interested.

Cheers,
Nick

=========

Grégory Joseph added a comment - 10/Nov/11 1:59 AM - edited

Thanks !
I'll follow the issue, but a non-generally available build wouldn't really at this point, since I'm working on our build infrastructure for the whole team. Of course I'll be happy to give it a spin if you want me to check it'd work for us.

=========

Marek Parfianowicz added a comment - 23/Jan/13 12:11 AM

I believe that it was solved in https://jira.atlassian.com/browse/CLOV-1064 (3.1.6 release).