Uploaded image for project: 'Clover'
  1. Clover
  2. CLOV-1064

Lack of Filtering/Reporting Consistency Across Maven Goals

    XMLWordPrintable

Details

    Description

      The various filtering options (statement contexts, method contexts, exclusions, etc) do not seem to be applied consistently across the various clover2 maven goals: clover2:clover, clover2:check: clover2:log. The discrepancies between clover2:clover and clover2:check essentially render clover2:check useless as it cannot be relied upon for accurate threshold reporting.

      The attached maven project demonstrates the following use cases:
      1. A method covered by tests
      2. A method not covered by tests
      3. A property method excluded via the property method context
      4. A constructor method excluded via the constructor method context
      5. A method filtered using a custom method context
      6. A statement filtered using a custom statement context
      7. A file filtered using the "exclusion" configuration option

      In order to execute the example, unzip the attached zip file and run the following:

      mvn clean clover2:setup test clover2:aggregate clover2:clover clover2:check clover2:log

      Things to note:

      Clover2:clover generates correctly. Coverage is at 50% and 71.4% of the code is correctly filtered out.

      Clover2:check reports coverage at 66.7%. Now in this case clover2:check is reporting coverage higher than clover2:clover. I'm not sure how it's getting there, but in my production examples the opposite tends to be the case: clover2:check reports lower coverage than clover2:clover.

      Clover2:log reports coverage at 33.3%:

      [INFO] Clover Coverage Report
      Coverage Timestamp: Fri Mar 02 12:26:09 EST 2012

      Coverage Overview -
      Coverage:-
      Methods: 2/7 (28.6%)
      Statements: 4/12 (33.3%)
      Branches: 1/2 (50%)
      Total: 33.3%
      Complexity:-
      Avg Method: 1.1428572
      Density: 0.6666667
      Total: 8

      If I change my maven execution to the following:

      mvn clean clover2:instrument test clover2:aggregate clover2:clover clover2:check clover2:log

      where clover2:setup is swapped out for clover2:instrument, the coverage check matches the report: 50%.

      As you can see, none of these tools seems to be reporting the same thing. You might say "Leo, why not use clover2:instrument so at least clover2:clover and clover:2check" match? Well, there's a problem with clover2:instrument as well: http://forums.atlassian.com/thread.jspa?threadID=22387

      I've been using clover for some time now (~5+ years). Your main competitive advantage over the other guys (jacoco, cobertura, emma, etc) is your advanced filtering and reporting capabilities. The frustrating thing is that in these 5 years none of these features have ever seemed to work consistently across maven goals nor in integration with other tools like STS and Sonar.

      Please make it a priority to ensure that these features are enabled consistently and tested for as part of every release. You know, ensure you have proper code coverage on your code coverage product.

      Please feel free to integrate my example project into your automated regression suite!

      Thanks!

      Attachments

        1. clover-3.1.6-snapshot-v2.z01
          5.80 MB
        2. clover-3.1.6-snapshot-v2.zip
          5.07 MB
        3. clover-example.zip
          11 kB
        4. clover-example.zip
          7 kB
        5. output.log
          33 kB

        Activity

          People

            mstudman Michael Studman (Inactive)
            97463bf031d0 Leo Hart
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Time Tracking

                Estimated:
                Original Estimate - 4h
                4h
                Remaining:
                Remaining Estimate - 4h
                4h
                Logged:
                Time Spent - Not Specified
                Not Specified