Uploaded image for project: 'JIRA Service Desk Server'
  1. JIRA Service Desk Server
  2. JSDSERVER-5736

Poor performance with high CPU and a high number of SdOffThreadEventJobRunner threads

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Highest
    • Resolution: Fixed
    • Affects Version/s: 3.9.0, 3.9.1, 3.9.2, 3.9.3, 3.9.4, 3.9.6, 3.9.7, 3.9.8, 3.9.9, 3.9.10, 3.9.11, 3.10.0, 3.10.1, 3.10.2, 3.11.0, 3.11.1, 3.16.0, 4.0.0, 4.1.0, 4.2.0
    • Fix Version/s: 4.3.0
    • Component/s: SLA

      Description

      JSD 3.9.0 attempts to address some of the friction between the SLA system and automation (JSDSERVER-4743) and poor issue creation performance by introducing a wrapper event type (inspired by OnCommitEvent) and an “expectation” system.

      The expectation system gives features that are interested in one or more eligible event types a way to explicitly define the work that should be done before a wrapped event is dispatched, by submitting “jobs” that are executed in the strict cluster-wide order of their submission (no more than one job at a time for each issue) using a thread pool to avoid blocking any request threads (though we just use the submitting thread if it’s not a request thread).

      The wrapper event type does the same for the work that should be done after what we refer to as “completion”, by defining @EventListener methods of type public void(ServiceDeskWrappedOnCompletionEvent).

      At least two recent support cases have involved severe performance degradation of a node in and/or the database for an instance that seems to have been caused or exacerbated by the expectation system, so we’ll link potential causes to this issue as we find them.

      Diagnosis

      • High CPU usage on DB server
      • Increased number of threads used by the Jira process
      • High number of SdOffThreadEventJobRunner threads on thread dumps connecting to the database

      Possible workaround (JSD 3.9+)

      These steps affect the expectation system such that jobs are always executed immediately on the submitting thread, without touching any OffThreadEventJobRunner or PSMQ code paths, as if the submitting threads are never request threads (JSDSERVER-5730).

      1. Go to the dark feature settings page (<baseURL>/secure/SiteDarkFeatures!default.jspa)
      2. Remove the feature flag sd.internal.base.off.thread.on.completion.events.enabled, if it exists
      3. Add the following feature flag: sd.internal.base.off.thread.on.completion.events.disabled

      SLA accuracy shouldn’t be negatively affected, but issue creation might take longer as a result. WHEN issue created automation rules with SLA-related JQL should still work (JSDSERVER-4743).

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              mchandra@atlassian.com Mohil Chandra
              Reporter:
              dazabani Delan Azabani (Inactive)
              Votes:
              181 Vote for this issue
              Watchers:
              157 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: