Reindex triggered though REST API doesn't stop NodeReindexServiceThread

XMLWordPrintable

    • 6.04
    • 15
    • Severity 2 - Major
    • 60

      Summary

      When you initiate full locked reindexing through REST API /rest/api/2/reindex?type=FOREGROUND, nodeReindexServiceThread:thread-1 at the same node still tries to read operations from replicatedindexoperation and fails. This spams the logs and lead to data inconsistency.

      Environment

      • 2 JIRA DC nodes

      Steps to Reproduce

      1. Start both JIRA nodes
      2. Initiate full locked reindex at Node1: 
        curl  -X POST http://127.0.0.1:8091/rest/api/2/reindex?type=FOREGROUND
      3. Modify some issues at Node2

      Expected Results

      Node1 will ignore messages in replicatedindexoperation table and does do index reconciliation.

      Actual Results

      Thread nodeReindexServiceThread:thread1 tries to read messages from replicatedindexoperation table and you can see following in the logs:

      2016-11-22 17:48:29,105 NodeReindexServiceThread:thread-1 ERROR      [c.a.j.issue.index.DefaultIndexManager] Wait attempt timed out - waited 30000 milliseconds
com.atlassian.jira.issue.index.IndexException: Wait attempt timed out - waited 30000 milliseconds
        at com.atlassian.jira.issue.index.DefaultIndexManager.obtain(DefaultIndexManager.java:794)
        at com.atlassian.jira.issue.index.DefaultIndexManager.access$600(DefaultIndexManager.java:88)
        at com.atlassian.jira.issue.index.DefaultIndexManager$IndexLock.tryLock(DefaultIndexManager.java:1118)
        at com.atlassian.jira.issue.index.DefaultIndexManager.getIndexLock(DefaultIndexManager.java:780)
        at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:533)
        at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssueObjects(DefaultIndexManager.java:438)
        ... 3 filtered
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:22)
        at com.sun.proxy.$Proxy11.reIndexIssueObjects(Unknown Source)
        at com.atlassian.jira.index.ha.DefaultNodeReindexService.updateIssueIndex(DefaultNodeReindexService.java:404)
        at com.atlassian.jira.index.ha.DefaultNodeReindexService.updateAffectedIndexes(DefaultNodeReindexService.java:298)
        at com.atlassian.jira.index.ha.DefaultNodeReindexService.reIndex(DefaultNodeReindexService.java:252)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2016-11-22 17:48:29,106 NodeReindexServiceThread:thread-1 ERROR      [c.a.j.issue.index.DefaultIndexManager] Could not reindex: com.atlassian.jira.issue.util.IssueObjectIssuesIterable (2 items): [PRG-4, PRG-6]

      Please note nodeReindexServiceThread:thread1 doesn't have any chance to obtain the lock as Reindex is running at the same time:

      2016-11-22 17:48:28,462 IssueIndexer:thread-9 INFO admin     [c.a.j.r.v2.index.ReindexResource] Re-indexing is 99% complete. Current index: Issue

      Notes

      None

      Workaround

      Don't use REST API and use JIRA UI to initiate the reindexing.

            Assignee:
            Tomasz Zwierzchowski
            Reporter:
            Andriy Yakovlev [Atlassian]
            Votes:
            27 Vote for this issue
            Watchers:
            25 Start watching this issue

              Created:
              Updated:
              Resolved: