-
Bug
-
Resolution: Fixed
-
Highest (View bug fix roadmap)
-
8.10.0, 8.11.0, 8.12.0, 8.13.0, 8.14.0, 8.13.3, 8.15.0, 8.13.5
-
8.1
-
43
-
Severity 2 - Major
-
479
-
Issue Summary
It is possible that background indexing tasks (like Background indexing, SLA recalculation and others) may fill up the indexer queue and affect user create/update issue/comment operation by:
- slowing down create/update issue/comment user request operations
- timing out user indexing requests (IndexException: Wait attempt timed out...) which will results in an error shown after 30sec to users and may result in changes not being visible index
There are couple of factors which will result in increasing the possibility of having this problem when triggering background indexing:
- fast Lucene document creation
- poor IO performance - this will slow down conditional updates, i.e. inserting the Lucene document in the index
With the above we will have a background indexing task(s) which may produce Lucene Documents faster than the indexer can consume them. This will fill up the indexer queue which is also used by user create/update tasks (HTTP requests).
Steps to Reproduce
TODO
Expected Results
Background indexing tasks should not affect user indexing tasks.
Actual Results
Currently a background indexing task can fill up the indexer queue and user indexing requests will timeout after 30sec:
/secure/CreateIssueDetails.jspa [c.a.j.issue.index.DefaultIndexManager] Wait attempt timed out - waited 30000 milliseconds com.atlassian.jira.issue.index.IndexException: Wait attempt timed out - waited 30000 milliseconds at com.atlassian.jira.issue.index.DefaultIndexManager.obtain(DefaultIndexManager.java:872) at com.atlassian.jira.issue.index.DefaultIndexManager.await(DefaultIndexManager.java:843) at com.atlassian.jira.issue.index.DefaultIndexManager.executeWithIndexLock(DefaultIndexManager.java:830) at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:618) at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssueObjects(DefaultIndexManager.java:494) at com.atlassian.jira.issue.index.DefaultIndexManager.reIndex(DefaultIndexManager.java:540) at com.atlassian.jira.issue.index.DefaultIndexManager.reIndex(DefaultIndexManager.java:523)
Workaround
In Jira DC:
- Background indexing should not be used. Always favour foreground indexing on one of the nodes which will not accept any user requests during re-indexing. This will guarantee faster re-indexing and not affecting concurrent user requests.
- SLA recalculation: the number of recalculation threads should be limited to 1
In Jira Server:
- IO performance should be investigated: conditional index updates trigger index flush which performance can be negatively affected by slow disk: KB: Troubleshooting performance with JIRA Stats
- Specifically check: Disk performance, updateDocumentsWithVersionMillis, processDBRMessageUpdateWithRelatedIndexInMillis
- Background indexing should not be triggered on a regular basis. Since Jira 8.10 the index should be consistent with the DB and there is no need to background indexing to guarantee this.
- If re-indexing is needed always favour foreground indexing (in Jira Server this means Jira being not available for users). Note that Jira DC provides HA in case of running foreground indexing.
Note that a slightly slower background indexing time is expected (since Jira 8.10) and this it not the problem described by this issue. This issue is about the problem of background indexing tasks taking over the indexer and affecting all user requests. Explained here: KB: Background reindex is slow after upgrading to Jira 8.10 and later
Note that the indexer may be flooded by apps triggering re-indexing in non request (user) threads. This may be very similar to running a full foreground re-index (multi-thread) with parallel user requests. In such case the plugin app should not trigger re-indexing on more then one thread. The workaround should include checking if there is a way of controlling the number of re-indexing threads.
- is duplicated by
-
-
- Closed
-
- is related to
-
-
- Closed
-
-
-
- Closed
-
-
-
- Closed
-
-
JRASERVER-70422 Create Lucene document based replication for Jira DC for index replication
- Closed
-
GHS-210293 Loading...
-
-
-
- relates to
-
-
- Closed
-
-
- Closed
- mentioned in
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
[JRASERVER-72045] IndexException: Wait attempt timed out - waited 30000 milliseconds caused by background indexing tasks
e68278496e06 you may be interested in this fix: https://jira.atlassian.com/browse/JRASERVER-73874
Should be back-ported to 8.20.x soon.
Still getting this error on 8.13.6.
The error is related to ScriptRunner scripted custom fields using Search Provider class.
due upgrade from v7 to v8 , the searchProvider class was replaced by searchService.
even the scripts was updated to use new method, there are several errors while re-indexing those fields.
2021-11-16 11:47:20,193-0300 IssueIndexer:thread-5 ERROR <-USER-> 706x2926x1 ndh56w 105.210.50.127 /secure/admin/IndexReIndex!reindex.jspa [c.o.scriptrunner.customfield.GroovyCustomField] Script field failed on issue: <-ISSUE->, field: <-FIELD-> com.atlassian.jira.issue.index.SearchUnavailableException at com.atlassian.jira.issue.index.DefaultIndexManager.getEntitySearcher(DefaultIndexManager.java:927) at com.atlassian.jira.issue.index.DefaultIndexManager.getIssueSearcher(DefaultIndexManager.java:906) at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:38) at com.sun.proxy.$Proxy17.getIssueSearcher(Unknown Source) at com.atlassian.jira.issue.search.SearchProviderFactoryImpl.getSearcher(SearchProviderFactoryImpl.java:17) at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.getIssueSearcher(LuceneSearchProvider.java:132) at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.searchDocuments(LuceneSearchProvider.java:455) at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.runSearch(LuceneSearchProvider.java:450) at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.getHits(LuceneSearchProvider.java:230) at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:374) at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:137) at com.atlassian.jira.issue.search.providers.LuceneSearchProvider.search(LuceneSearchProvider.java:142) at com.atlassian.jira.bc.issue.search.DefaultSearchService.search(DefaultSearchService.java:118) at com.atlassian.jira.bc.issue.search.SearchService$search$0.call(Unknown Source) at Script25.run(Script25.groovy:71) 2021-11-16 11:47:20,237-0300 IssueIndexer:thread-21 ERROR <-USER-> 706x2926x1 ndh56w 105.210.50.127 /secure/admin/IndexReIndex!reindex.jspa [c.a.j.issue.index.DefaultIndexManager] Wait attempt timed out - waited 30000 milliseconds
jasonsmith1590948304 - please create a support case so we can investigate your problem.
We're on 8.13.9 and are still getting this during a background reindex.
^ What is up with that display name? :DDD
Anyhow there's plenty of teams affected and as Atlassian has it's process on how fast they release that particular version obviously the teams responsible for their instances of the product have their processes to upgrade, obviously on both sides it takes time.
We use LTS releases only for example. We do not upgrade each time there is release. It all depends team to team and/or impact/benefits.
That seems to be not a great solution. Update of a Jira Production site is not done in just a few days.
jutamat.phothisitthisak1 I would recommend upgrading to 8.16.x where the issue is fixed.
We also face this issue at the moment with Jira DC. v 8.14 and also we have co-issue by having project re-indexing in the same time as background indexing which cause the node has been stuck and lockdown.. and we still waiting for Atlassian response how to fix this issue.
Thanks , we would like to go as soon as possible to 8.13.6. 
Still waiting with experiencing problem.
> We upgraded to 8.14
ilona.cypers@hlag.com - the fix was done in 8.16.0 (released) and only now back-ported to 8.13.6 (not released yet);
Currently (at the time of writing this comment) the only version of Jira with this bug fixed is 8.16.0.
Hello!
We upgraded to 8.14 two weeks ago and still have this issue with massive performance degredation during the time of background reindex.
Wish we knew this prior to upgrading to 8.13. Users upgrading to 8.x needs to be warned about this bug.
a6b647410708 - I would expect 8.13.6 to be released around end of April / beginning of May.
@maciej can you please let us know the tentative release of 8.13.6 ??
We are planning for upgrade to 8.13 but want to wait for this fix
mswinarski could you share when will be backported to 8.13.x,
we are experiencing that problem
KB about new Jira indexing queues: https://confluence.atlassian.com/display/JIRAKB/Jira+Indexing+Queue
In JIRA 8.16 we are going to release the following solution for this bug. We are introducing 2 priority queues to the single threaded index updater (per index):
- the primary indexing queue - accepts only indexing requests created by user requests (HTTP threads)
- the secondary indexing queue - accepts all other indexing requests (created by non-request threads)
Indexing operations from the secondary queue are only consumed when the primary queue is empty. With this change user actions (triggering re-indexing) should not be affected by background indexing tasks like: Jira background indexing, SLA recalculations, DVCS indexing, index synchronisation (Jira DC), etc.
Both queues may have internal priority, however currently this is an internal API and only core Jira indexing background processes (node index replay and DBR) have higher priority then all other indexing tasks. In future we may expose the priority API as a public API. For now plugins triggering any re-indexing in user request or in background threads are not required to change anything.
We have also improved significantly the performance of all conditional updates (since Jira 8.10 this refers to all index updates except foreground re-index) by decreasing the frequency of requesting a fresh searcher (this operation triggers an index flush) on the index. This should just speed up background indexing. How much? This depends on many factors:
- traffic shape - how many concurrent updates are done when any background indexing task is running
- IO performance - index flush triggers a disk operation so with this change slower IO should not affect indexing performance so much
With changes above we have introduced some new log stats which describe the state of both queues in an aggregated form (to be described).
We are planning to backport this fix to 8.13.x (exact version - to be decided).
Hi e5e6470d1f18,
Thanks for the heads-up! Although you didn't provide the KB URL, I believe that you might be referring to Background reindex is slow after upgrading to Jira 8.10 and later, so I added a note at the top of the KB with a link to this bug.
Cheers!
It would be great if this ticket was linked on the KB about slower re-indexing to save customers time hunting for this issue. As the reporter has mentioned, while background indexes running for 20% doesn't really impact UX, this issue renders the background index feature pointless on our instance.
With a globally used Jira Server instance it means re-indexing is either really disruptive to a group of users, or has to wait for weekends - also not really desirable given a re-index is sometimes required to make the instance perform as expected when adding new fields etc.
The bottleneck was the database in our cases of Jira 8.20.x and 9.4 on 2 vCPU-machine with 32 G RAM and 25 indexing threads. For some reason, the indexing ran a query from Atlassian Greenhopper, that was very DB CPU-intensive and slow. For 12 hours the full indexing had completed 9% with that query:
with plan:
created the index:
*create index "AO_60DB71_LEXORANK_FLD_ISS_TYP" ON "AO_60DB71_LEXORANK" ("FIELD_ID", "ISSUE_ID", "TYPE")
the load dropped and the plan now is:
And also the query:
I added the index:
*create index "AO_FC7135_DATA_ROW_issue_field_pos_id" on "AO_FC7135_DATA_ROW" ("FIELD_CONFIG_ID", "ISSUE_ID", "ORDER_POSITION", "ID");
and the plan now is:
With these two database indexes, the full indexing of JIRA completed in 2.5 hours.
The slow full index in JIRA might also be caused by inefficient DB queries. Consider adding (these) DB indexes and only then upgrade machines and indexing threads.