Status: Closed (View Workflow)
Affects Version/s: 8.10.0, 8.11.0, 8.12.0, 8.13.0, 8.14.0, 8.13.3, 8.15.0
Introduced in Version:8.1
Support reference count:43
Symptom Severity:Severity 2 - Major
Bug Fix Policy:
It is possible that background indexing tasks (like Background indexing, SLA recalculation and others) may fill up the indexer queue and affect user create/update issue/comment operation by:
- slowing down create/update issue/comment user request operations
- timing out user indexing requests (IndexException: Wait attempt timed out...) which will results in an error shown after 30sec to users and may result in changes not being visible index
There are couple of factors which will result in increasing the possibility of having this problem when triggering background indexing:
- fast Lucene document creation
- poor IO performance - this will slow down conditional updates, i.e. inserting the Lucene document in the index
With the above we will have a background indexing task(s) which may produce Lucene Documents faster than the indexer can consume them. This will fill up the indexer queue which is also used by user create/update tasks (HTTP requests).
Background indexing tasks should not affect user indexing tasks.
Currently a background indexing task can fill up the indexer queue and user indexing requests will timeout after 30sec:
In Jira DC:
- Background indexing should not be used. Always favour foreground indexing on one of the nodes which will not accept any user requests during re-indexing. This will guarantee faster re-indexing and not affecting concurrent user requests.
- SLA recalculation: the number of recalculation threads should be limited to 1
In Jira Server:
- IO performance should be investigated: conditional index updates trigger index flush which performance can be negatively affected by slow disk: KB: Troubleshooting performance with JIRA Stats
- Specifically check: Disk performance, updateDocumentsWithVersionMillis, processDBRMessageUpdateWithRelatedIndexInMillis
- Background indexing should not be triggered on a regular basis. Since Jira 8.10 the index should be consistent with the DB and there is no need to background indexing to guarantee this.
- If re-indexing is needed always favour foreground indexing (in Jira Server this means Jira being not available for users). Note that Jira DC provides HA in case of running foreground indexing.
Note that a slightly slower background indexing time is expected (since Jira 8.10) and this it not the problem described by this issue. This issue is about the problem of background indexing tasks taking over the indexer and affecting all user requests. Explained here: KB: Background reindex is slow after upgrading to Jira 8.10 and later
Note that the indexer may be flooded by apps triggering re-indexing in non request (user) threads. This may be very similar to running a full foreground re-index (multi-thread) with parallel user requests. In such case the plugin app should not trigger re-indexing on more then one thread. The workaround should include checking if there is a way of controlling the number of re-indexing threads.