History-based SLA selectively fails to pause/stop metrics on status transitions in JSM DC multi-node clusters due to a watermark over-advance race condition

XMLWordPrintable

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Medium
    • 11.3.10, 12.0.0, 10.3.24
    • Affects Version/s: 10.3.13
    • Component/s: SLA
    • 5
    • Severity 2 - Major
    • 22

      Issue Summary

      In a multi-node JSM DC cluster, the history-based SLA calculation engine selectively fails to pause or stop specific SLA metrics during a status transition, while correctly evaluating other SLA metrics for the same issue, same transition, same timestamp.

      The root cause is a race condition in SlaDataManagerImpl where the per-issue watermark (sla.last.processed.dateTime) is advanced to raw now() instead of the latest change-history timestamp actually observed. Under concurrent multi-node processing, one node's event processing writes the watermark before a status transition whose change item has been created earlier but becomes visible slightly later. The transition then falls below the watermark and is excluded from every future gated-window rescan (watermark, now] — the ongoing metric never pauses/stops.

      Key Observations:

      • Metrics that must start (idle/empty timeline) perform a full-history recompute and react correctly.
      • Metrics that must pause/stop (already ongoing/non-empty timeline) use the gated window (watermark, now] and miss the transition once the watermark over-advances past it
      • Unrelated events (e.g., attachment add) trigger catch-up because they write a new watermark that may re-expose the missed transition
      • sla.last.processed.dateTime updates to the attachment timestamp, not the status transition timestamp
      • ~50% permanent-miss rate reproduced on a 2-node JSM DC 10.3.23 cluster

      Steps to Reproduce

      1. Set up a JSM Data Center cluster with 2+ nodes
      1. Configure a JSM project with multiple SLA metrics — at least one that must start on a status and another that must pause/stop on the same status (e.g., [Internal] Testing starts on "Testing"; [Internal] Implementation pauses on "Testing")
      1. Generate concurrent load across nodes (or transition the same issue from different nodes in quick succession)
      1. Transition an issue to "Testing" status
      1. Observe that [Internal] Testing SLA starts correctly (full recompute path), but [Internal] Implementation does NOT pause (gated window path)
      1. Verify via REST: GET /rest/api/2/issue/{key}/properties/sla.last.processed.dateTime — the watermark timestamp will NOT correspond to the status transition time
      1. Add an attachment to the issue — observe that the SLA processor "wakes up" and catches up on the missed transition

      Expected Results

      All SLA metrics should correctly evaluate every status transition regardless of which node processes the event. The per-issue watermark should never advance past a change that has not been evaluated by all ongoing metrics.

      Actual Results

      • Ongoing metrics (non-empty timeline) miss status transitions when the watermark over-advances past the change-history timestamp
      • The missed transition is permanently excluded from future rescans
      • Only a full reconstruction (which forces a full-history recompute) corrects the metric
      • The issue recurs on new tickets under a continued multi-node load

      Workaround

      Reconstruct/recalculate the SLA on each affected issue:
      {{}}

      POST /rest/servicedesk/1/servicedesk/sla/admin/task/destructive/reconstruct/jql Body: {"jql": "issue in (ABC-12345, ABC-54321, ABC-78910)"}

      {{}}
      This clears the SLA timeline and forces a full-history recompute that sees the missed transition.

      Caveat: This is a repair, not a prevention — the race recurs on new tickets under multi-node load.

              Assignee:
              Khair-un-Nisa Iqbal
              Reporter:
              skavatekar
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: