Details
-
Bug
-
Resolution: Fixed
-
Highest
-
3.9.0, 3.9.1, 3.9.2, 3.9.3, 3.9.4, 3.9.6, 3.9.7, 3.9.8, 3.9.9, 3.9.10, 3.9.11, 3.10.0, 3.10.1, 3.10.2, 3.11.0, 3.11.1, 3.16.0, 3.16.1, 4.0.0, 4.1.0, 4.2.0
-
207
-
Severity 2 - Major
-
1,218
-
Description
Hi everyone,
Thank you for your feedback on the ticket and supporting our team in our investigation!
After analysing the problem, we have identified the issue of "Poor performance with high CPU and a high number of SdOffThreadEventJobRunner threads" has been fixed in JSD 4.4.0. This problem occurred due to the unbounded nature of threads before 4.4.0, which resulted in a high DB load on the instance.
However - we have identified two other issues with the JSM async processing logic which we need to resolve, related to the problems reported in this bug ticket. These issues are tracked in their respective tickets -
- This is related to deadlocking of threads when there are frequent actions on one request. A fix for this is released behind a dark feature in 4.9.0
- The development team will be working on enabling this by default in future. More details on the fix -
https://confluence.atlassian.com/jirakb/deadlocking-in-jira-service-desk-when-frequently-updating-the-same-issue-979428323.html
- This is related to threads deadlocking when processing for one issue takes over 5 minutes. This ticket is currently gathering impact.
Please let us know if you have any further concerns with the above, please open a support ticket via https://support.atlassian.com
Thank you,
Alex
Description
JSD 3.9.0 attempts to address some of the friction between the SLA system and automation (JSDSERVER-4743) and poor issue creation performance by introducing a wrapper event type (inspired by OnCommitEvent) and an “expectation” system.
The expectation system gives features that are interested in one or more eligible event types a way to explicitly define the work that should be done before a wrapped event is dispatched, by submitting “jobs” that are executed in the strict cluster-wide order of their submission (no more than one job at a time for each issue) using a thread pool to avoid blocking any request threads (though we just use the submitting thread if it’s not a request thread).
The wrapper event type does the same for the work that should be done after what we refer to as “completion”, by defining @EventListener methods of type public void(ServiceDeskWrappedOnCompletionEvent).
At least two recent support cases have involved severe performance degradation of a node in and/or the database for an instance that seems to have been caused or exacerbated by the expectation system, so we’ll link potential causes to this issue as we find them.
Diagnosis
- High CPU usage on DB server
- Increased number of threads used by the Jira process
- High number of SdOffThreadEventJobRunner threads on thread dumps connecting to the database
Possible workaround (JSD 3.9+)
These steps affect the expectation system such that jobs are always executed immediately on the submitting thread, without touching any OffThreadEventJobRunner or PSMQ code paths, as if the submitting threads are never request threads (JSDSERVER-5730).
- Go to the dark feature settings page (<baseURL>/secure/SiteDarkFeatures!default.jspa)
- Remove the feature flag sd.internal.base.off.thread.on.completion.events.enabled, if it exists
- Add the following feature flag: sd.internal.base.off.thread.on.completion.events.disabled
- Restart JIRA
SLA accuracy shouldn’t be negatively affected, but issue creation might take longer as a result. WHEN issue created automation rules with SLA-related JQL should still work (JSDSERVER-4743).
Attachments
Issue Links
- is caused by
-
-
- Closed
-
-
-
- Closed
-
- is related to
-
-
- Closed
-
- blocks
-
GHS-143296 Loading...
- causes
-
-
- is cloned by
-
-
- mentioned in
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-