-
Bug
-
Resolution: Fixed
-
Highest
-
3.9.0, (20)
3.9.1, 3.9.2, 3.9.3, 3.9.4, 3.9.6, 3.9.7, 3.9.8, 3.9.9, 3.9.10, 3.9.11, 3.10.0, 3.10.1, 3.10.2, 3.11.0, 3.11.1, 3.16.0, 3.16.1, 4.0.0, 4.1.0, 4.2.0
-
207
-
Severity 2 - Major
-
1,218
-
Hi everyone,
Thank you for your feedback on the ticket and supporting our team in our investigation!
After analysing the problem, we have identified the issue of "Poor performance with high CPU and a high number of SdOffThreadEventJobRunner threads" has been fixed in JSD 4.4.0. This problem occurred due to the unbounded nature of threads before 4.4.0, which resulted in a high DB load on the instance.
However - we have identified two other issues with the JSM async processing logic which we need to resolve, related to the problems reported in this bug ticket. These issues are tracked in their respective tickets -
- This is related to deadlocking of threads when there are frequent actions on one request. A fix for this is released behind a dark feature in 4.9.0
- The development team will be working on enabling this by default in future. More details on the fix -
https://confluence.atlassian.com/jirakb/deadlocking-in-jira-service-desk-when-frequently-updating-the-same-issue-979428323.html
- This is related to threads deadlocking when processing for one issue takes over 5 minutes. This ticket is currently gathering impact.
Please let us know if you have any further concerns with the above, please open a support ticket via https://support.atlassian.com
Thank you,
Alex
Description
JSD 3.9.0 attempts to address some of the friction between the SLA system and automation (JSDSERVER-4743) and poor issue creation performance by introducing a wrapper event type (inspired by OnCommitEvent) and an “expectation” system.
The expectation system gives features that are interested in one or more eligible event types a way to explicitly define the work that should be done before a wrapped event is dispatched, by submitting “jobs” that are executed in the strict cluster-wide order of their submission (no more than one job at a time for each issue) using a thread pool to avoid blocking any request threads (though we just use the submitting thread if it’s not a request thread).
The wrapper event type does the same for the work that should be done after what we refer to as “completion”, by defining @EventListener methods of type public void(ServiceDeskWrappedOnCompletionEvent).
At least two recent support cases have involved severe performance degradation of a node in and/or the database for an instance that seems to have been caused or exacerbated by the expectation system, so we’ll link potential causes to this issue as we find them.
Diagnosis
- High CPU usage on DB server
- Increased number of threads used by the Jira process
- High number of SdOffThreadEventJobRunner threads on thread dumps connecting to the database
Possible workaround (JSD 3.9+)
These steps affect the expectation system such that jobs are always executed immediately on the submitting thread, without touching any OffThreadEventJobRunner or PSMQ code paths, as if the submitting threads are never request threads (JSDSERVER-5730).
- Go to the dark feature settings page (<baseURL>/secure/SiteDarkFeatures!default.jspa)
- Remove the feature flag sd.internal.base.off.thread.on.completion.events.enabled, if it exists
- Add the following feature flag: sd.internal.base.off.thread.on.completion.events.disabled
- Restart JIRA
SLA accuracy shouldn’t be negatively affected, but issue creation might take longer as a result. WHEN issue created automation rules with SLA-related JQL should still work (JSDSERVER-4743).
- is caused by
-
-
- Closed
-
-
-
- Closed
-
- is related to
-
-
- Closed
-
- blocks
-
GHS-143296 You do not have permission to view this issue
- causes
-
-
ITOPS-1449 Loading...
- is cloned by
-
-
- mentioned in
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
[JSDSERVER-5736] Poor performance with high CPU and a high number of SdOffThreadEventJobRunner threads
We experiencing a crash that seems to be related to this. (Service Management Server 4.15)
There is contention on the Queue and Message tables which seems to cause either regular Deadlock or sometimes results in a MySQL crash.
Aug 19 07:05:10 localhost mysqld[11332]: Some pointers may be invalid and cause the dump to abort.
Aug 19 07:05:10 localhost mysqld[11332]: Query (0x7f34040107c0): update `AO_319474_MESSAGE` set `CLAIMANT` = null, `CLAIMANT_TIME` = null where `AO_319474_MESSAGE`.`QUEUE_ID` = 1214021 and `AO_319474_MESSAGE`.`CLAIMANT` is not null
Aug 19 07:05:10 localhost mysqld[11332]: Connection ID (thread ID): 12046760
Hi everyone,
Thank you for your feedback regarding this bug and bringing it to our attention. Since this bug has been re-opened in May we have resumed our investigation into this issue.
Currently we're unable to reproduce this issue and are looking to work with customers directly affected who are able to reproduce the problem.
If you're able to reproduce the issue, can you please open a support ticket via https://support.atlassian.com, and let the support engineer know about this ticket and this message. We'll work with you to make sure the development team has the information they need to begin working on resolving it.
Please let us know if you have any concerns with the above.
Thank you,
Alex
To be clear, I am not on the Jira Development team, so I can't update the fix version, and I can't fix this problem myself. We are experiencing this on an internal Jira that my team owns, so I have reopened it for the Jira team to re-triage.
Reopening this for the Jira team to triage again, as it seems to still be occurring for a lot of folks in later versions.
Curious what the workaround would do for those not running Service Desk. Is there still a benefit of adding sd.internal.base.off.thread.on.completion.events.disabled?
Magic to see in the logs of JSD DC on 4.13.3.
And flag has been removed, maybe other cause ?
Is there any fix for 3.16.1 ? or is it only to upgrade past 4.3.0 version of JSD? WHat is the exact action causing this? SLA processing? Reporting. Im trying to understand what the trigger is as it comes out of nowhere.
JSD Enterprise release (3.16.x) don't have this fix?? My customer need this fix for his versione (3.16.x)
Does anyone who has upgraded to the fix version have feedback on performance post-upgrade? I'm waiting for the enterprise release, but am curious about others' experience.
Is there any particular kind of activity that would trigger this bug? And is it only constrained to activity within a Service Desk project?
Unfortunately im quite sure they will not fix it anytime soon, if they havent done it when they changed major version and did an overhaul of lots of the inner workings they wont do it now. Most likely the problem is much too complicated to solve and while the systems still work, why change it?
I hope im wrong, as we have the same issue in our logs everyday and we have a very very small instance of 6 agents and at most 30 customers at any one time on the platform.
Hello All,
We need a solution for this bug with a certain urgency, plz look at us atlassian!
Luca, this issue you describe looks like JRASERVER-63002 which should be solved in 8.0.0, but still seeing it in 8.0.2 as well ...
Sill occurring in JSD 3.16.1, also with workarounds suggested, slowing a lot Jira response time. Users and DBA are not happy 
Following query is using all CPU in DB server:
SELECT CG.ID, CG.issueid, CG.AUTHOR, CG.CREATED, CI.ID, CI.groupid, CI.FIELDTYPE, CI.FIELD, CI.OLDVALUE, CI.OLDSTRING, CI.NEWVALUE, CI.NEWSTRING FROM jiraschema.changegroup CG INNER JOIN jiraschema.changeitem CI ON CG.ID = CI.groupid WHERE CG.issueid=@P0 AND CI.FIELD=@P1 ORDER BY CG.CREATED ASC, CI.ID ASC
We are an Atlassian partner in Brazil and we have a big customer that is being terribly impacted by this bug.
After this error the customer is raising questions about the viability of continuing to use the JIRA Service Desk. We need a solution urgently.
We are experiencing this issue as well.
Jira Software: 7.12.0/JSD 3.15.0
Log scan results report this issue every day
Jira Core 7.12.3 / JIRA Service Desk 3.15.3
Has there been any movement on this? My customer continues to experience this issue
We had to reenable ServiceDeskWrappedOnCompletionEvents this afternoon. Our users were experiencing extreme slowness when creating issues in the portal. Reenabling the events has caused the db connections to spike again so we've had to resort to toggling the events on and off when we've noticed high db connection usage
I've been experiencing this issue with a customer since we upgraded to JSD 3.10.2 at the end of February. I discovered this bug yesterday and we definitely meet all of the criteria in the diagnosis section. We implemented the workaround above and the Connection Pool graphs show the difference we've seen in connection usage after disabling the queue.
Hi everyone,
Thank you for your feedback on the ticket and supporting our team in our investigation!
After analysing the problem, we have identified the issue of "Poor performance with high CPU and a high number of SdOffThreadEventJobRunner threads" has been fixed in JSD 4.4.0. This problem occurred due to the unbounded nature of threads before 4.4.0, which resulted in a high DB load on the instance.
However - we have identified two other issues with the JSM async processing logic which we need to resolve, related to the problems reported in this bug ticket. These issues are tracked in their respective tickets -
1.
JSDSERVER-5730https://confluence.atlassian.com/jirakb/deadlocking-in-jira-service-desk-when-frequently-updating-the-same-issue-979428323.html
2.
JSDSERVER-8635Please let us know if you have any further concerns with the above, please open a support ticket via https://support.atlassian.com
Thank you,
Alex