Uploaded image for project: 'Clover'
  1. Clover
  2. CLOV-418

Investigate how best to support per-test coverage for tests run in parallel.



    • Our product teams collect and evaluate feedback from a number of different sources. To learn more about how we use customer feedback in the planning process, check out our new feature policy.


      CPARTY natter on the topic:

      mstudman: multi-processing == multi-threaded test execution?
      npellow: not so sure. I assumed so.
      mstudman: would be good to know. we should have something in place (thread-local slice coverage) so that we at least support it even if not very performant which we can more easily do now with my per-test coverage refactoring
      npellow: yes! was thinking that. looking good btw!
      mstudman: if we did have to support this, we'd probably be better off changing instrumented code so that the thread-local slice coverage was fetched once per per method call (of the instrumented app) and stored as a local to minimise thread-local slowness
      npellow: TestNG already supports multi-threaded execution of tests, which is how bamboo run their tests i believe
      mstudman: there may even be some hotspot mojo benefit to storing as a local ordinarily.
      mstudman: i'll raise a JIRA with my thoughts and we can think about doing it when there's a need
      npellow: ah, so you mean only store per-test coverage at the method level?
      npellow: and not for each line?
      mstudman: no
      mstudman: i mean, a call to recorder.inc(0) will, if we have thread-local slice coverage, mean a call to a thread local for every element - performance FAIL
      mstudman: if we at least cache the thread-local as a local variable of the instrumented method we avoid a lot of these thread-local calls
      mstudman: so # thread-local calls == # method calls in test run
      npellow: wouldn't lead to spurious results?
      mstudman: not # thread-local calls == # elements in test run
      mstudman: how so?
      npellow: if two tests enter the same method at the same time and both see the same thread local?
      mstudman: impossible
      mstudman: two tests, same method => two threads => two thread locals
      npellow: 1 cache == 1 bug
      mstudman: sorry, to back up: we would have 1 x slice-coverage per thread
      mstudman: no caches here
      mstudman: not a cache in the conventional sense
      npellow: k, gotcha. so many thread locals would be cached at the start of each method
      mstudman: there would only ever be one local variable defined per instrumented method which would, as local variables do, live on the stack associated with the executing thread
      npellow: k
      npellow: i see
      mstudman: it would be a tradeoff
      mstudman: currently we have 1...N slice recorders in play but normally only 1 - and they record coverage across all threads of the app
      mstudman: with my suggestion, we would have 1 x slice recorder per thread but they would only record coverage by their associated thread
      mstudman: so if they spawned a thread the coverage of that spawned thread would be unattributed to a test (unless we used thread groups or something - not clear how this might work)




            Unassigned Unassigned
            mstudman Michael Studman (Inactive)
            1 Vote for this issue
            1 Start watching this issue