Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-59386

JIRA Datacenter node failed to restore the snapshot from another node

    XMLWordPrintable

Details

    Description

      Summary

      JIRA Datacenter nod - failed to restore the snapshot from another node

      Environment

      • JIRA Datacenter 6.4.12
      • 2 Nodes

      Steps to Reproduce

      1. Start index replication at Node2 from Node1
      2. Watch the status in logs

      Expected Results

      Index is replicated and Node2 is operational
      Message in logs:

      2016-01-12 01:35:17,456 NodeReindexServiceThread:thread-1 INFO      [jira.index.ha.DefaultIndexRecoveryManager] Recovering search indexes - 100% complete...
      2016-01-12 01:35:17,483 NodeReindexServiceThread:thread-1 INFO      [jira.index.ha.DefaultIndexCopyService] Index restore complete
      

      Actual Results

      Index is not replicated and Node2 is not operational
      The below exception is thrown in the log file:

      2016-01-12 01:34:24,945 NodeReindexServiceThread:thread-1 ERROR      [jira.index.ha.DefaultNodeReindexService] Error re-indexing node changes
      java.lang.RuntimeException: java.io.IOException: Destination '/apps/opt/atlassian/application-data/jira/caches/indexes/issues' already exists
      	at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager$ReplaceIndexRunner.run(DefaultIndexRecoveryManager.java:374)
      	at com.atlassian.jira.issue.index.DefaultIndexManager.withReindexLock(DefaultIndexManager.java:382)
      	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at com.atlassian.util.profiling.object.ObjectProfiler.profiledInvoke(ObjectProfiler.java:83)
      	at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:28)
      	at com.sun.proxy.$Proxy17.withReindexLock(Unknown Source)
      	at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager.recoverIndexFromBackup(DefaultIndexRecoveryManager.java:124)
      	at com.atlassian.jira.index.ha.DefaultIndexCopyService$MessageConsumer.restoreIndex(DefaultIndexCopyService.java:175)
      	at com.atlassian.jira.index.ha.DefaultIndexCopyService.restoreIndex(DefaultIndexCopyService.java:79)
      	at com.atlassian.jira.index.ha.DefaultNodeReindexService.updateAffectedIndexes(DefaultNodeReindexService.java:344)
      	at com.atlassian.jira.index.ha.DefaultNodeReindexService.reIndex(DefaultNodeReindexService.java:291)
      	at com.atlassian.jira.index.ha.DefaultNodeReindexService.access$000(DefaultNodeReindexService.java:58)
      	at com.atlassian.jira.index.ha.DefaultNodeReindexService$1.run(DefaultNodeReindexService.java:82)
      	(...)
      Caused by: java.io.IOException: Destination '/apps/opt/atlassian/application-data/jira/caches/indexes/issues' already exists
      	at org.apache.commons.io.FileUtils.moveDirectory(FileUtils.java:1739)
      	at org.apache.commons.io.FileUtils.moveDirectoryToDirectory(FileUtils.java:1781)
      	at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager$ReplaceIndexRunner.replaceIndexes(DefaultIndexRecoveryManager.java:452)
      	at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager$ReplaceIndexRunner.run(DefaultIndexRecoveryManager.java:370)
      	... 22 more
      

      Notes

      Index restore also initiated automatically when one of nodes finishes full re-index and asks other nodes to apply new index.

      Workaround

      Restart Node so it will pickup latest valid Index snapshot from other nodes

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              ayakovlev@atlassian.com Andriy Yakovlev [Atlassian]
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: