Uploaded image for project: 'Clover'
  1. Clover
  2. CLOV-418

Investigate how best to support per-test coverage for tests run in parallel.

    • Our product teams collect and evaluate feedback from a number of different sources. To learn more about how we use customer feedback in the planning process, check out our new feature policy.

      CPARTY natter on the topic:

      mstudman: multi-processing == multi-threaded test execution?
      npellow: not so sure. I assumed so.
      mstudman: would be good to know. we should have something in place (thread-local slice coverage) so that we at least support it even if not very performant which we can more easily do now with my per-test coverage refactoring
      npellow: yes! was thinking that. looking good btw!
      mstudman: if we did have to support this, we'd probably be better off changing instrumented code so that the thread-local slice coverage was fetched once per per method call (of the instrumented app) and stored as a local to minimise thread-local slowness
      npellow: TestNG already supports multi-threaded execution of tests, which is how bamboo run their tests i believe
      mstudman: there may even be some hotspot mojo benefit to storing as a local ordinarily.
      mstudman: i'll raise a JIRA with my thoughts and we can think about doing it when there's a need
      npellow: ah, so you mean only store per-test coverage at the method level?
      npellow: and not for each line?
      mstudman: no
      mstudman: i mean, a call to recorder.inc(0) will, if we have thread-local slice coverage, mean a call to a thread local for every element - performance FAIL
      mstudman: if we at least cache the thread-local as a local variable of the instrumented method we avoid a lot of these thread-local calls
      mstudman: so # thread-local calls == # method calls in test run
      npellow: wouldn't lead to spurious results?
      mstudman: not # thread-local calls == # elements in test run
      mstudman: how so?
      npellow: if two tests enter the same method at the same time and both see the same thread local?
      mstudman: impossible
      mstudman: two tests, same method => two threads => two thread locals
      npellow: 1 cache == 1 bug
      mstudman: sorry, to back up: we would have 1 x slice-coverage per thread
      mstudman: no caches here
      mstudman: not a cache in the conventional sense
      npellow: k, gotcha. so many thread locals would be cached at the start of each method
      mstudman: there would only ever be one local variable defined per instrumented method which would, as local variables do, live on the stack associated with the executing thread
      npellow: k
      npellow: i see
      mstudman: it would be a tradeoff
      mstudman: currently we have 1...N slice recorders in play but normally only 1 - and they record coverage across all threads of the app
      mstudman: with my suggestion, we would have 1 x slice recorder per thread but they would only record coverage by their associated thread
      mstudman: so if they spawned a thread the coverage of that spawned thread would be unattributed to a test (unless we used thread groups or something - not clear how this might work)

            [CLOV-418] Investigate how best to support per-test coverage for tests run in parallel.

            thanks a lot for the hint, we will try it! which version of clover is the first one supporting the option?

            rupert THURNER added a comment - thanks a lot for the hint, we will try it! which version of clover is the first one supporting the option?

            Rupert,

            You can try supplying the undocumented (and unsupported but "works for me") setting -Dclover.pertest.coverage.threading=synchronized when running your unit tests. This way you will get accurate per-test coverage information at the cost of a bit of performance.

            Michael.

            Michael Studman (Inactive) added a comment - - edited Rupert, You can try supplying the undocumented (and unsupported but "works for me") setting -Dclover.pertest.coverage.threading=synchronized when running your unit tests. This way you will get accurate per-test coverage information at the cost of a bit of performance. Michael.

            multithreaded testing is now a feature of maven as well, see http://weblogs.java.net/blog/johnsmart/archive/2010/07/06/running-junit-tests-parallel-maven

            rupert THURNER added a comment - multithreaded testing is now a feature of maven as well, see http://weblogs.java.net/blog/johnsmart/archive/2010/07/06/running-junit-tests-parallel-maven

              Unassigned Unassigned
              mstudman Michael Studman (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: