Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-63284

Reindex triggered though REST API doesn't stop NodeReindexServiceThread

XMLWordPrintable

      Summary

      When you initiate full locked reindexing through REST API /rest/api/2/reindex?type=FOREGROUND, nodeReindexServiceThread:thread-1 at the same node still tries to read operations from replicatedindexoperation and fails. This spams the logs and lead to data inconsistency.

      Environment

      • 2 JIRA DC nodes

      Steps to Reproduce

      1. Start both JIRA nodes
      2. Initiate full locked reindex at Node1: 
        curl  -X POST http://127.0.0.1:8091/rest/api/2/reindex?type=FOREGROUND
      3. Modify some issues at Node2

      Expected Results

      Node1 will ignore messages in replicatedindexoperation table and does do index reconciliation.

      Actual Results

      Thread nodeReindexServiceThread:thread1 tries to read messages from replicatedindexoperation table and you can see following in the logs:

      2016-11-22 17:48:29,105 NodeReindexServiceThread:thread-1 ERROR      [c.a.j.issue.index.DefaultIndexManager] Wait attempt timed out - waited 30000 milliseconds
com.atlassian.jira.issue.index.IndexException: Wait attempt timed out - waited 30000 milliseconds
        at com.atlassian.jira.issue.index.DefaultIndexManager.obtain(DefaultIndexManager.java:794)
        at com.atlassian.jira.issue.index.DefaultIndexManager.access$600(DefaultIndexManager.java:88)
        at com.atlassian.jira.issue.index.DefaultIndexManager$IndexLock.tryLock(DefaultIndexManager.java:1118)
        at com.atlassian.jira.issue.index.DefaultIndexManager.getIndexLock(DefaultIndexManager.java:780)
        at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:533)
        at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssueObjects(DefaultIndexManager.java:438)
        ... 3 filtered
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:22)
        at com.sun.proxy.$Proxy11.reIndexIssueObjects(Unknown Source)
        at com.atlassian.jira.index.ha.DefaultNodeReindexService.updateIssueIndex(DefaultNodeReindexService.java:404)
        at com.atlassian.jira.index.ha.DefaultNodeReindexService.updateAffectedIndexes(DefaultNodeReindexService.java:298)
        at com.atlassian.jira.index.ha.DefaultNodeReindexService.reIndex(DefaultNodeReindexService.java:252)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)
2016-11-22 17:48:29,106 NodeReindexServiceThread:thread-1 ERROR      [c.a.j.issue.index.DefaultIndexManager] Could not reindex: com.atlassian.jira.issue.util.IssueObjectIssuesIterable (2 items): [PRG-4, PRG-6]

      Please note nodeReindexServiceThread:thread1 doesn't have any chance to obtain the lock as Reindex is running at the same time:

      2016-11-22 17:48:28,462 IssueIndexer:thread-9 INFO admin     [c.a.j.r.v2.index.ReindexResource] Re-indexing is 99% complete. Current index: Issue

      Notes

      None

      Workaround

      Don't use REST API and use JIRA UI to initiate the reindexing.

            [JRASERVER-63284] Reindex triggered though REST API doesn't stop NodeReindexServiceThread

            set-jac-bot made changes -
            Fixed in Enterprise Release/s New: [Download 7.13, 7.6|https://confluence.atlassian.com/enterprise/atlassian-enterprise-releases-948227420.html]
            Kevin Allen made changes -
            Remote Link New: This issue links to "Page (Confluence)" [ 478052 ]
            Vivek Yadavayyanamath made changes -
            Affects Version/s New: 7.12.3 [ 81711 ]
            Billy McCarthy made changes -
            Remote Link New: This issue links to "Page (Confluence)" [ 459904 ]
            Bugfix Automation Bot made changes -
            Minimum Version New: 6.04
            Tomasz Zwierzchowski made changes -
            Remote Link New: This issue links to "Page (Confluence)" [ 413444 ]
            Krzysztof Kepinski (Inactive) made changes -
            Fix Version/s New: 7.13.1 [ 81709 ]
            Fix Version/s Original: 7.13.0 [ 81291 ]
            Tomasz Zwierzchowski made changes -
            Resolution New: Fixed [ 1 ]
            Status Original: Waiting for Release [ 12075 ] New: Closed [ 6 ]
            SET Analytics Bot made changes -
            UIS Original: 53 New: 60
            SET Analytics Bot made changes -
            Support reference count Original: 14 New: 15
            UIS Original: 43 New: 53

              tzwierzchowski Tomasz Zwierzchowski
              ayakovlev@atlassian.com Andriy Yakovlev [Atlassian]
              Affected customers:
              27 This affects my team
              Watchers:
              25 Start watching this issue

                Created:
                Updated:
                Resolved: