Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-59386

JIRA Datacenter node failed to restore the snapshot from another node

XMLWordPrintable

      Summary

      JIRA Datacenter nod - failed to restore the snapshot from another node

      Environment

      • JIRA Datacenter 6.4.12
      • 2 Nodes

      Steps to Reproduce

      1. Start index replication at Node2 from Node1
      2. Watch the status in logs

      Expected Results

      Index is replicated and Node2 is operational
      Message in logs: 

      2016-01-12 01:35:17,456 NodeReindexServiceThread:thread-1 INFO      [jira.index.ha.DefaultIndexRecoveryManager] Recovering search indexes - 100% complete...
2016-01-12 01:35:17,483 NodeReindexServiceThread:thread-1 INFO      [jira.index.ha.DefaultIndexCopyService] Index restore complete

      Actual Results

      Index is not replicated and Node2 is not operational
      The below exception is thrown in the log file:

      2016-01-12 01:34:24,945 NodeReindexServiceThread:thread-1 ERROR      [jira.index.ha.DefaultNodeReindexService] Error re-indexing node changes
java.lang.RuntimeException: java.io.IOException: Destination '/apps/opt/atlassian/application-data/jira/caches/indexes/issues' already exists
	at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager$ReplaceIndexRunner.run(DefaultIndexRecoveryManager.java:374)
	at com.atlassian.jira.issue.index.DefaultIndexManager.withReindexLock(DefaultIndexManager.java:382)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.atlassian.util.profiling.object.ObjectProfiler.profiledInvoke(ObjectProfiler.java:83)
	at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:28)
	at com.sun.proxy.$Proxy17.withReindexLock(Unknown Source)
	at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager.recoverIndexFromBackup(DefaultIndexRecoveryManager.java:124)
	at com.atlassian.jira.index.ha.DefaultIndexCopyService$MessageConsumer.restoreIndex(DefaultIndexCopyService.java:175)
	at com.atlassian.jira.index.ha.DefaultIndexCopyService.restoreIndex(DefaultIndexCopyService.java:79)
	at com.atlassian.jira.index.ha.DefaultNodeReindexService.updateAffectedIndexes(DefaultNodeReindexService.java:344)
	at com.atlassian.jira.index.ha.DefaultNodeReindexService.reIndex(DefaultNodeReindexService.java:291)
	at com.atlassian.jira.index.ha.DefaultNodeReindexService.access$000(DefaultNodeReindexService.java:58)
	at com.atlassian.jira.index.ha.DefaultNodeReindexService$1.run(DefaultNodeReindexService.java:82)
	(...)
Caused by: java.io.IOException: Destination '/apps/opt/atlassian/application-data/jira/caches/indexes/issues' already exists
	at org.apache.commons.io.FileUtils.moveDirectory(FileUtils.java:1739)
	at org.apache.commons.io.FileUtils.moveDirectoryToDirectory(FileUtils.java:1781)
	at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager$ReplaceIndexRunner.replaceIndexes(DefaultIndexRecoveryManager.java:452)
	at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager$ReplaceIndexRunner.run(DefaultIndexRecoveryManager.java:370)
	... 22 more

      Notes

      Index restore also initiated automatically when one of nodes finishes full re-index and asks other nodes to apply new index.

      Workaround

      Restart Node so it will pickup latest valid Index snapshot from other nodes

              Unassigned Unassigned
              ayakovlev@atlassian.com Andriy Yakovlev [Atlassian]
              Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

                Created:
                Updated:
                Resolved: