With off-threading enabled, AO_319474_MESSAGE database table is growing and most SdSerialisedOffThreadProcessor threads are stuck waiting for cluster lock.
Checking the clusterlockstatus database table, the cluster locks held by the Jira node are associated to issues with substantial amount of comments (~5000 comments) and the MESSAGE_COUNT for those issues are high.
This can happen when SLA events are being processed off thread with database backed events.
Update issue comment that trigger SLA update whilst issues are being created by multiple concurrent users.
Relevant SLA and automation are updated.
SLA and automation are not updated.
Most SdSerialisedOffThreadProcessor threads are in cluster lock wait, e.g:
One option is to disable off thread processing for SLA events. This is not always recommended, as it will result in SLAs being calculated on the HTTP threads so users may notice slower response times on the UI.
- Go to the dark feature settings page (<baseURL>/secure/SiteDarkFeatures!default.jspa)
- Remove the feature flag sd.internal.base.off.thread.on.completion.events.enabled, if it exists
- Remove the feature flag sd.internal.base.db.backed.completion.events.enabled, if it exists
- Add the following feature flag: sd.internal.base.off.thread.on.completion.events.disabled
- Add the following feature flag: sd.internal.bounded.off.thread.on.completion.events.disabled
- Restart JIRA
A more complete fix, which improves the way that cluster locks are handled with off-threading enabled is available in 4.21.0, enabled by default, and also in Jira 4.13.15+, and 4.20.3+, enabled by setting dark features (ensuring any dark features ending with on.completion.events.disabled are removed):