-
Bug
-
Resolution: Fixed
-
Low
-
6.4.14, 7.1.10, 7.2.3, 7.12.3
-
6.04
-
15
-
Severity 2 - Major
-
60
-
Summary
When you initiate full locked reindexing through REST API /rest/api/2/reindex?type=FOREGROUND, nodeReindexServiceThread:thread-1 at the same node still tries to read operations from replicatedindexoperation and fails. This spams the logs and lead to data inconsistency.
Environment
- 2 JIRA DC nodes
Steps to Reproduce
- Start both JIRA nodes
- Initiate full locked reindex at Node1:
curl -X POST http://127.0.0.1:8091/rest/api/2/reindex?type=FOREGROUND - Modify some issues at Node2
Expected Results
Node1 will ignore messages in replicatedindexoperation table and does do index reconciliation.
Actual Results
Thread nodeReindexServiceThread:thread1 tries to read messages from replicatedindexoperation table and you can see following in the logs:
2016-11-22 17:48:29,105 NodeReindexServiceThread:thread-1 ERROR [c.a.j.issue.index.DefaultIndexManager] Wait attempt timed out - waited 30000 milliseconds
com.atlassian.jira.issue.index.IndexException: Wait attempt timed out - waited 30000 milliseconds
at com.atlassian.jira.issue.index.DefaultIndexManager.obtain(DefaultIndexManager.java:794)
at com.atlassian.jira.issue.index.DefaultIndexManager.access$600(DefaultIndexManager.java:88)
at com.atlassian.jira.issue.index.DefaultIndexManager$IndexLock.tryLock(DefaultIndexManager.java:1118)
at com.atlassian.jira.issue.index.DefaultIndexManager.getIndexLock(DefaultIndexManager.java:780)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:533)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssueObjects(DefaultIndexManager.java:438)
... 3 filtered
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:22)
at com.sun.proxy.$Proxy11.reIndexIssueObjects(Unknown Source)
at com.atlassian.jira.index.ha.DefaultNodeReindexService.updateIssueIndex(DefaultNodeReindexService.java:404)
at com.atlassian.jira.index.ha.DefaultNodeReindexService.updateAffectedIndexes(DefaultNodeReindexService.java:298)
at com.atlassian.jira.index.ha.DefaultNodeReindexService.reIndex(DefaultNodeReindexService.java:252)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-11-22 17:48:29,106 NodeReindexServiceThread:thread-1 ERROR [c.a.j.issue.index.DefaultIndexManager] Could not reindex: com.atlassian.jira.issue.util.IssueObjectIssuesIterable (2 items): [PRG-4, PRG-6]
Please note nodeReindexServiceThread:thread1 doesn't have any chance to obtain the lock as Reindex is running at the same time:
2016-11-22 17:48:28,462 IssueIndexer:thread-9 INFO admin [c.a.j.r.v2.index.ReindexResource] Re-indexing is 99% complete. Current index: Issue
Notes
None
Workaround
Don't use REST API and use JIRA UI to initiate the reindexing.