-
Bug
-
Resolution: Fixed
-
Low (View bug fix roadmap)
-
6.4.14, 7.2.3, 7.1.10, 7.12.3
-
6.04
-
15
-
Severity 2 - Major
-
60
-
Summary
When you initiate full locked reindexing through REST API /rest/api/2/reindex?type=FOREGROUND, nodeReindexServiceThread:thread-1 at the same node still tries to read operations from replicatedindexoperation and fails. This spams the logs and lead to data inconsistency.
Environment
- 2 JIRA DC nodes
Steps to Reproduce
- Start both JIRA nodes
- Initiate full locked reindex at Node1:
curl -X POST http://127.0.0.1:8091/rest/api/2/reindex?type=FOREGROUND - Modify some issues at Node2
Expected Results
Node1 will ignore messages in replicatedindexoperation table and does do index reconciliation.
Actual Results
Thread nodeReindexServiceThread:thread1 tries to read messages from replicatedindexoperation table and you can see following in the logs:
2016-11-22 17:48:29,105 NodeReindexServiceThread:thread-1 ERROR [c.a.j.issue.index.DefaultIndexManager] Wait attempt timed out - waited 30000 milliseconds
com.atlassian.jira.issue.index.IndexException: Wait attempt timed out - waited 30000 milliseconds
at com.atlassian.jira.issue.index.DefaultIndexManager.obtain(DefaultIndexManager.java:794)
at com.atlassian.jira.issue.index.DefaultIndexManager.access$600(DefaultIndexManager.java:88)
at com.atlassian.jira.issue.index.DefaultIndexManager$IndexLock.tryLock(DefaultIndexManager.java:1118)
at com.atlassian.jira.issue.index.DefaultIndexManager.getIndexLock(DefaultIndexManager.java:780)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:533)
at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssueObjects(DefaultIndexManager.java:438)
... 3 filtered
at java.lang.reflect.Method.invoke(Method.java:498)
at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:22)
at com.sun.proxy.$Proxy11.reIndexIssueObjects(Unknown Source)
at com.atlassian.jira.index.ha.DefaultNodeReindexService.updateIssueIndex(DefaultNodeReindexService.java:404)
at com.atlassian.jira.index.ha.DefaultNodeReindexService.updateAffectedIndexes(DefaultNodeReindexService.java:298)
at com.atlassian.jira.index.ha.DefaultNodeReindexService.reIndex(DefaultNodeReindexService.java:252)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
2016-11-22 17:48:29,106 NodeReindexServiceThread:thread-1 ERROR [c.a.j.issue.index.DefaultIndexManager] Could not reindex: com.atlassian.jira.issue.util.IssueObjectIssuesIterable (2 items): [PRG-4, PRG-6]
Please note nodeReindexServiceThread:thread1 doesn't have any chance to obtain the lock as Reindex is running at the same time:
2016-11-22 17:48:28,462 IssueIndexer:thread-9 INFO admin [c.a.j.r.v2.index.ReindexResource] Re-indexing is 99% complete. Current index: Issue
Notes
None
Workaround
Don't use REST API and use JIRA UI to initiate the reindexing.
[JRASERVER-63284] Reindex triggered though REST API doesn't stop NodeReindexServiceThread
The workaround isn't exactly ideal because we have to first SSH onto a JIRA node to stop a monitoring process and than afterwards we are able to run a full re-index. This unfortunately adds a lot of overhead to our team especially since only a small group of people have access to do that and we need to perform a re-index at least once a week due to our sheer size. Hopefully this can be bumped up in priority because it's most certainly needed.
This issue blocks the ability to effectively automate this process and would require human resources to do this manually over the weekend during our maintenance hours. No wants that! 
Please consider taking this on priority as automating this process is crucial for Adobe.
Any chance this is on the radar?.. Having data center and the ability to index when every you want on a node is awesome but i think the need to trigger it from REST successfully is still needed..
Does this lead to index inconsistency on all the DC nodes? Or just the node that is performing the full re-index?