-
Bug
-
Resolution: Fixed
-
Medium
-
4.13.0
-
3
-
Severity 1 - Critical
-
45
-
Issue Summary
When SLA configuration changes all issues in the project may be re-indexed.
The re-index is done in a heavy way (real-time cross-cluster re-index) and can affect an unlimited number of issues.
This can cause the whole instance to fail.
Steps to Reproduce
- Create a project with >50K issues on a multinode DC instance
- Change SLA configuration in the project
Expected Results
SLAs on the project should be updated and no performance impact
Actual Results
A huge number of indexing calls cause high CPU load and slow down the whole instance. Other indexing task may start to timeout (DBR - real time index replication, NodeReindexService - delayed node index replication/index consistency check, ... any other re-indexing triggered by background, ie. non-http threads). This may cause index inconsistency between nodes which can eventually be only resolved with a full foreground reindex.
Workaround
Currently there is no workaround for the problem.
- caused
-
HOT-96659 You do not have permission to view this issue
- mentioned in
-
Page Failed to load
-
Page Failed to load
-
Page Failed to load
-
Page Failed to load
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
- PIR - Medium Priority Action
-
PIR-10219 Loading...
- relates to
-
DELTA-1137 Loading...
Form Name |
---|
Hello,
Thank you for your patience waiting for an update for this issue. JSM and Jira teams have investigated this issue and addressed the behaviours that contributed to this bug.
Root Cause
Excessive amount of reindexing calls for a large amount of issues in a short period of time may cause the indexing queue overflow.
JSM Footprint
When an SLA configuration is updated or a new SLA is created, JSM will fetch all the issues affected by the change, calculate the new SLA value, save it to the DB, and then reindex all changed issues. More on how we recalculate SLAs can be found here. For JSM to have a significant footprint on indexes, the project will have to have 50k+ tickets, and a large portion of them needs to be affected by the configuration change which is in most scenarios not the case as JSM would only recalculate ongoing cycles, and in general scenario the majority of historical tickets would have the SLAs completed. Nonetheless, there are scenarios where such changes are made.
A side note here: in our testing, we have found that sheer number of issues is not enough to overload the indexing queues. Issues with a significant amounts of worklog, history and/or comments need to be present in the affected project for this issue to manifest itself.
Contributing factors
Mitigation options
Product improvements