Details
-
Bug
-
Resolution: Duplicate
-
Low
-
6.4.12, 7.1.0
-
6.04
-
Description
Summary
JIRA Datacenter nod - failed to restore the snapshot from another node
Environment
- JIRA Datacenter 6.4.12
- 2 Nodes
Steps to Reproduce
- Start index replication at Node2 from Node1
- Watch the status in logs
Expected Results
Index is replicated and Node2 is operational
Message in logs:
2016-01-12 01:35:17,456 NodeReindexServiceThread:thread-1 INFO [jira.index.ha.DefaultIndexRecoveryManager] Recovering search indexes - 100% complete... 2016-01-12 01:35:17,483 NodeReindexServiceThread:thread-1 INFO [jira.index.ha.DefaultIndexCopyService] Index restore complete
Actual Results
Index is not replicated and Node2 is not operational
The below exception is thrown in the log file:
2016-01-12 01:34:24,945 NodeReindexServiceThread:thread-1 ERROR [jira.index.ha.DefaultNodeReindexService] Error re-indexing node changes java.lang.RuntimeException: java.io.IOException: Destination '/apps/opt/atlassian/application-data/jira/caches/indexes/issues' already exists at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager$ReplaceIndexRunner.run(DefaultIndexRecoveryManager.java:374) at com.atlassian.jira.issue.index.DefaultIndexManager.withReindexLock(DefaultIndexManager.java:382) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.atlassian.util.profiling.object.ObjectProfiler.profiledInvoke(ObjectProfiler.java:83) at com.atlassian.jira.config.component.SwitchingInvocationHandler.invoke(SwitchingInvocationHandler.java:28) at com.sun.proxy.$Proxy17.withReindexLock(Unknown Source) at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager.recoverIndexFromBackup(DefaultIndexRecoveryManager.java:124) at com.atlassian.jira.index.ha.DefaultIndexCopyService$MessageConsumer.restoreIndex(DefaultIndexCopyService.java:175) at com.atlassian.jira.index.ha.DefaultIndexCopyService.restoreIndex(DefaultIndexCopyService.java:79) at com.atlassian.jira.index.ha.DefaultNodeReindexService.updateAffectedIndexes(DefaultNodeReindexService.java:344) at com.atlassian.jira.index.ha.DefaultNodeReindexService.reIndex(DefaultNodeReindexService.java:291) at com.atlassian.jira.index.ha.DefaultNodeReindexService.access$000(DefaultNodeReindexService.java:58) at com.atlassian.jira.index.ha.DefaultNodeReindexService$1.run(DefaultNodeReindexService.java:82) (...) Caused by: java.io.IOException: Destination '/apps/opt/atlassian/application-data/jira/caches/indexes/issues' already exists at org.apache.commons.io.FileUtils.moveDirectory(FileUtils.java:1739) at org.apache.commons.io.FileUtils.moveDirectoryToDirectory(FileUtils.java:1781) at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager$ReplaceIndexRunner.replaceIndexes(DefaultIndexRecoveryManager.java:452) at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager$ReplaceIndexRunner.run(DefaultIndexRecoveryManager.java:370) ... 22 more
Notes
Index restore also initiated automatically when one of nodes finishes full re-index and asks other nodes to apply new index.
Workaround
Restart Node so it will pickup latest valid Index snapshot from other nodes
Attachments
Issue Links
- duplicates
-
JRASERVER-59721 Index corruption when restoring index from snapshot
- Closed