[CLOV-1015] Test Optimization says 0 tests should run but all tests are run anyway

Type: Bug
Resolution: Fixed
Priority: High
Fix Version/s: 3.1.6
Affects Version/s: 3.1.0
Component/s: Maven plugin
Labels:
None
Environment:

Java 6, Maven 2.2.1, Maven Surefire plugin 2.9,

Recently upgraded my Maven builds to use the Maven Surefire plugin version 2.9. Now my builds that run Clover2:optimized and say 0 tests should run, run all tests anyway. The upshot of this is that our optimized test runs are now taking over an hour versus the 15 to 20 minutes they were when the tests were correctly optimized away.

Here is a section of the log file that shows the problem:

    [shell] cd dba; mvn -PPERSONAL,UnitTests,Clover.optimize -Djava.awt.headless=true --offline -Dforkmode=once -Dmaven.download.meter=silent -DskipAssembly -Dmaven.test.failure.ignore=true -Dmaven.test.redirectTestOutputToFile=false -Dmaven.clover.licenseLocation=/prodinfolocal/BambooHome/clover.license -Dclover.plugin.version=3.1.0 -Dclover.version=3.1.0 install  clover2:aggregate clover2:clover
[INFO] 
NOTE: Maven is executing in offline mode. Any artifacts not already in your local
repository will be inaccessible.

[INFO] Scanning for projects...
[WARNING] 
	Profile with id: 'PERSONAL' has not been activated.

[WARNING] 
	Profile with id: 'UnitTests' has not been activated.

[INFO] ------------------------------------------------------------------------
[INFO] Building dba
[INFO]    task-segment: [install]
[INFO] ------------------------------------------------------------------------
[INFO] [clover2:setup {execution: clover}]
[INFO] No Clover instrumentation done on source files in: [/Users/mhusby/workarea/dba/src/main/java] as no matching sources files found
[INFO] No Clover instrumentation done on source files in: [/Users/mhusby/workarea/dba/src/test/java] as no matching sources files found
[WARNING] Using [Squid:configuration:jar:1.0-SNAPSHOT], built on Fri Jul 22 15:44:02 EDT 2011 even though a Clovered version exists but it's older (lastModified: Fri Jul 22 15:43:02 EDT 2011 ) and could fail the build. Please consider running Clover again on that dependency's project.
[INFO] [resources:resources {execution: default-resources}]
[WARNING] Using platform encoding (MacRoman actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 2 resources
[INFO] Copying 1 resource
[INFO] [compiler:compile {execution: default-compile}]
[INFO] Nothing to compile - all classes are up to date
[INFO] [resources:testResources {execution: default-testResources}]
[WARNING] Using platform encoding (MacRoman actually) to copy filtered resources, i.e. build is platform dependent!
[INFO] Copying 6 resources
[INFO] [compiler:testCompile {execution: default-testCompile}]
[INFO] Nothing to compile - all classes are up to date
[INFO] [clover2:optimize {execution: clover}]
[INFO] Adding fileset: directory=/Users/mhusby/workarea/dba/target/clover/src-test-instrumented, includes=[**/*Tests.*, **/*Test.*, **/Test*.*, qxjShouldNeverMatchAClass], excludes=[**/*TestCase.*, **/Test*Servlet.*, **/Abstract*.*, **/*FormModuleTests.*, **/*.xml, **/*$*.*]
[INFO] Clover estimates having saved around 44 seconds on this optimized test run. The full test run takes approx. 44 seconds
[INFO] Clover included 0 test classes in this run (total # test classes : 22)
[INFO] [surefire:test {execution: default-test}]
[INFO] Surefire report directory: /Users/mhusby/workarea/dba/target/surefire-reports

-------------------------------------------------------
 T E S T S
-------------------------------------------------------
Running edu.mit.broad.prodinfo.dba.BlobDefTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.327 sec
Running edu.mit.broad.prodinfo.dba.LongDefTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec
Running edu.mit.broad.prodinfo.dba.ExceptionWrapperTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec
Running edu.mit.broad.prodinfo.dba.DoubleDefTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec
Running edu.mit.broad.prodinfo.dba.CacheMapTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.108 sec
Running edu.mit.broad.prodinfo.dba.OurDBsTest
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.596 sec
Running edu.mit.broad.prodinfo.dba.VArrayDefTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec
Running edu.mit.broad.prodinfo.dba.VArrayFormatterTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.006 sec
Running edu.mit.broad.prodinfo.dba.RowDefTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.311 sec
Running edu.mit.broad.prodinfo.dba.BigDecimalDefTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec
Running edu.mit.broad.prodinfo.dba.DoubleFormatterTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0 sec
Running edu.mit.broad.prodinfo.dba.DateFormatterTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.007 sec
Running edu.mit.broad.prodinfo.dba.LongFormatterTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec
Running edu.mit.broad.prodinfo.dba.ClobDefTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.004 sec
Running edu.mit.broad.prodinfo.dba.DBTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.778 sec
Running edu.mit.broad.prodinfo.dba.DataIteratorTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec
Running edu.mit.broad.prodinfo.dba.ProcedureDefTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.052 sec
Running edu.mit.broad.prodinfo.dba.VArrayListTest
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.003 sec
Running edu.mit.broad.prodinfo.dba.DBPoolTest
Tests run: 8, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 39.08 sec
Running edu.mit.broad.prodinfo.dba.StringDefTest
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.002 sec
Running edu.mit.broad.prodinfo.dba.DataDefIteratorTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.001 sec
Running edu.mit.broad.prodinfo.dba.RowTest
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.54 sec

Results :

Tests run: 49, Failures: 0, Errors: 0, Skipped: 1

[INFO] [clover2:snapshot {execution: clover}]
[INFO] No span specified, using span of: 48s
[INFO] Saving snapshot to: /Users/mhusby/.clover/Squid-dba/clover.snapshot
[INFO] Clover Version 3.1.0, built on May 31 2011 (build-821)
[INFO] Loaded from: /Users/mhusby/.m2/repository/com/cenqua/clover/clover/3.1.0/clover-3.1.0.jar
[INFO] Clover: Academic License registered to Broad/MIT Institute.
[INFO] Updating snapshot '/Users/mhusby/.clover/Squid-dba/clover.snapshot' against Clover database at '/Users/mhusby/workarea/dba/target/clover/clover.db'

mentioned in: Wiki Page Failed to load

Jacek Jaroczynski (Inactive) added a comment - 10/May/2013 12:08 PM - edited

After analysis it looks like it was fixed in Clover 3.1.6. It works fine and skips all tests if necessary.

Jacek Jaroczynski (Inactive) added a comment - 10/May/2013 12:08 PM - edited After analysis it looks like it was fixed in Clover 3.1.6. It works fine and skips all tests if necessary.

Jacek Jaroczynski (Inactive) added a comment - 30/Apr/2013 7:15 AM - edited

Does it mean that test optimization does not work with surefire 2.9-2.11 at all? What about newer surefire version?

Doesn't it work only if tests should be skipped (0 tests to run)? What in case if e.g. 1 test should be run, does it work fine in that case?

Do we know answers to that questions or should I investigate that?

Jacek Jaroczynski (Inactive) added a comment - 30/Apr/2013 7:15 AM - edited Does it mean that test optimization does not work with surefire 2.9-2.11 at all? What about newer surefire version? Doesn't it work only if tests should be skipped (0 tests to run)? What in case if e.g. 1 test should be run, does it work fine in that case? Do we know answers to that questions or should I investigate that?

Andrey Klochkov added a comment - 21/Sep/2012 10:19 PM

Here's a workaround I found. This plugin runs after Clover's done with optimization and sets "skipTests" property if the property "test" is an empty string.

          <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>groovy-maven-plugin</artifactId>
            <version>1.4</version>
            <executions>
              <execution>
                <phase>process-test-classes</phase>
                <goals>
                  <goal>execute</goal>
                </goals>
                <configuration>
                  <source>
                    def tests = project.properties['test'];
                    log.info("Tests to run: " + tests);
                    if (tests != null &amp;&amp; tests.length() == 0) {
                      log.info("No tests to run");
                      project.properties['skipTests'] = 'true';
                    }
                  </source>
                </configuration>
              </execution>
            </executions>
          </plugin>

Andrey Klochkov added a comment - 21/Sep/2012 10:19 PM Here's a workaround I found. This plugin runs after Clover's done with optimization and sets "skipTests" property if the property "test" is an empty string. <plugin> <groupId>org.codehaus.mojo</groupId> <artifactId>groovy-maven-plugin</artifactId> <version>1.4</version> <executions> <execution> <phase>process-test-classes</phase> <goals> <goal>execute</goal> </goals> <configuration> <source> def tests = project.properties[ 'test' ]; log.info( "Tests to run: " + tests); if (tests != null && tests.length() == 0) { log.info( "No tests to run" ); project.properties[ 'skipTests' ] = ' true ' ; } </source> </configuration> </execution> </executions> </plugin>

Erik Husby added a comment - 19/Dec/2011 6:38 PM

I had to go all the way back to surefire 2.7.1 to get the correct behavior in Clover.

It would be really nice to get this fixed in Clover since I am sure I had a good reason to upgrade Surefire although I did not make a note. I need to run more builds before I commit to rolling back the surefire version.

Erik Husby added a comment - 19/Dec/2011 6:38 PM I had to go all the way back to surefire 2.7.1 to get the correct behavior in Clover. It would be really nice to get this fixed in Clover since I am sure I had a good reason to upgrade Surefire although I did not make a note. I need to run more builds before I commit to rolling back the surefire version.

Nick added a comment - 18/Dec/2011 11:26 PM

Hi Erik, It appears that the version of the Maven-Surefire plugin you are using (2.9 - 2.11) is not working with Clover's test optimisation.

Are you able to roll back the version of the surefire plugin to a version that does work with Clover'r test optimisation until this issue is resolved ?

Cheers,
Nick

Nick added a comment - 18/Dec/2011 11:26 PM Hi Erik, It appears that the version of the Maven-Surefire plugin you are using (2.9 - 2.11) is not working with Clover's test optimisation. Are you able to roll back the version of the surefire plugin to a version that does work with Clover'r test optimisation until this issue is resolved ? Cheers, Nick

Erik Husby added a comment - 16/Dec/2011 9:23 PM

Problem is still occurring with Maven-Surefire-plugin Version 2.11.

Is anyone going to address this issue? It really has a negative impact on our builds.

Erik Husby added a comment - 16/Dec/2011 9:23 PM Problem is still occurring with Maven-Surefire-plugin Version 2.11. Is anyone going to address this issue? It really has a negative impact on our builds.

Erik Husby added a comment - 16/Sep/2011 2:22 PM

This is really impacting our Bamboo build server. Any idea when it can be dealt with?

Erik Husby added a comment - 16/Sep/2011 2:22 PM This is really impacting our Bamboo build server. Any idea when it can be dealt with?

Assignee:: Jacek Jaroczynski (Inactive)

Reporter:: Erik Husby

Affected customers:: 1 This affects my team

Watchers:: 4 Start watching this issue

Created:: 22/Jul/2011 8:18 PM

Updated:: 10/Jun/2015 10:43 AM

Resolved:: 13/May/2013 9:25 AM

Details

Description

Attachments

Issue Links

Forms

Activity

Collapse comment: Jacek Jaroczynski (Inactive) added a comment - 10/May/2013 12:08 PM, Edited by Jacek Jaroczynski - 10/May/2013 2:25 PM

Expand comment: Jacek Jaroczynski (Inactive) added a comment - 10/May/2013 12:08 PM, Edited by Jacek Jaroczynski - 10/May/2013 2:25 PM

Collapse comment: Jacek Jaroczynski (Inactive) added a comment - 30/Apr/2013 7:15 AM, Edited by Jacek Jaroczynski - 30/Apr/2013 10:25 AM

Expand comment: Jacek Jaroczynski (Inactive) added a comment - 30/Apr/2013 7:15 AM, Edited by Jacek Jaroczynski - 30/Apr/2013 10:25 AM

Collapse comment: Andrey Klochkov added a comment - 21/Sep/2012 10:19 PM

Expand comment: Andrey Klochkov added a comment - 21/Sep/2012 10:19 PM

Collapse comment: Erik Husby added a comment - 19/Dec/2011 6:38 PM

Expand comment: Erik Husby added a comment - 19/Dec/2011 6:38 PM

Collapse comment: Nick added a comment - 18/Dec/2011 11:26 PM

Expand comment: Nick added a comment - 18/Dec/2011 11:26 PM

Collapse comment: Erik Husby added a comment - 16/Dec/2011 9:23 PM

Expand comment: Erik Husby added a comment - 16/Dec/2011 9:23 PM

Collapse comment: Erik Husby added a comment - 16/Sep/2011 2:22 PM

Expand comment: Erik Husby added a comment - 16/Sep/2011 2:22 PM

People

Dates