IndexException: Wait attempt timed out - waited 30000 milliseconds caused by background indexing tasks

Issue Summary

It is possible that background indexing tasks (like Background indexing, SLA recalculation and others) may fill up the indexer queue and affect user create/update issue/comment operation by:

• slowing down create/update issue/comment user request operations 
• timing out user indexing requests (IndexException: Wait attempt timed out...) which will results in an error shown after 30sec to users and may result in changes not being visible index

There are couple of factors which will result in increasing the possibility of having this problem when triggering background indexing:

• fast Lucene document creation
• poor IO performance - this will slow down conditional updates, i.e. inserting the Lucene document in the index

With the above we will have a background indexing task(s) which may produce Lucene Documents faster than the indexer can consume them. This will fill up the indexer queue which is also used by user create/update tasks (HTTP requests).  

Steps to Reproduce

TODO

Expected Results

Background indexing tasks should not affect user indexing tasks.

Actual Results

Currently a background indexing task can fill up the indexer queue and user indexing requests will timeout after 30sec:

/secure/CreateIssueDetails.jspa [c.a.j.issue.index.DefaultIndexManager] Wait attempt timed out - waited 30000 milliseconds
com.atlassian.jira.issue.index.IndexException: Wait attempt timed out - waited 30000 milliseconds
at com.atlassian.jira.issue.index.DefaultIndexManager.obtain(DefaultIndexManager.java:872)
at com.atlassian.jira.issue.index.DefaultIndexManager.await(DefaultIndexManager.java:843)
at com.atlassian.jira.issue.index.DefaultIndexManager.executeWithIndexLock(DefaultIndexManager.java:830)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:618)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssueObjects(DefaultIndexManager.java:494)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndex(DefaultIndexManager.java:540)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndex(DefaultIndexManager.java:523) 

Workaround

In Jira DC:

• Background indexing should not be used. Always favour foreground indexing on one of the nodes which will not accept any user requests during re-indexing. This will guarantee faster re-indexing and not affecting concurrent user requests.
• SLA recalculation: the number of recalculation threads should be limited to 1 

In Jira Server:

• IO performance should be investigated: conditional index updates trigger index flush which performance can be negatively affected by slow disk: KB: Troubleshooting performance with JIRA Stats
  • Specifically check: Disk performance, updateDocumentsWithVersionMillis, processDBRMessageUpdateWithRelatedIndexInMillis
• Background indexing should not be triggered on a regular basis. Since Jira 8.10 the index should be consistent with the DB and there is no need to background indexing to guarantee this.
• If re-indexing is needed always favour foreground indexing (in Jira Server this  means Jira being not available for users). Note that Jira DC provides HA in case of running foreground indexing.

Note that a slightly slower background indexing time is expected (since Jira 8.10) and this it not the problem described by this issue. This issue is about the problem of background indexing tasks taking over the indexer and affecting all user requests. Explained here: KB: Background reindex is slow after upgrading to Jira 8.10 and later

Note that the indexer may be flooded by apps triggering re-indexing in non request (user) threads. This may be very similar to running a full foreground re-index (multi-thread) with parallel user requests. In such case the plugin app should not trigger re-indexing on more then one thread. The workaround should include checking if there is a way of controlling the number of re-indexing threads.