Index corruption when restoring index from snapshot

Diagnosis:

After restoring an index from a snapshot (backup) file, many issues are missing.

An error similar to the following may be present in the application logs:

Caused by: java.io.IOException: Destination '/jira-home/caches/indexes/issues' already exists
	at org.apache.commons.io.FileUtils.moveDirectory(FileUtils.java:1739)
	at org.apache.commons.io.FileUtils.moveDirectoryToDirectory(FileUtils.java:1781)
	at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager$ReplaceIndexRunner.replaceIndexes(DefaultIndexRecoveryManager.java:452)
	at com.atlassian.jira.index.ha.DefaultIndexRecoveryManager$ReplaceIndexRunner.run(DefaultIndexRecoveryManager.java:370)
	... 22 more

The same code path is triggered when nodes perform major index related work in JIRA clusters, such as after a node completes a reindex or an administrator initiates a node to node index copy.

Summary:

When JIRA is performing major operations on the Lucene index it puts in place a barrier to prevent simultaneous access to the index. This barrier was bypassed by a change introduced in JIRA 6.4. As a result the index may be written to with an incomplete record of issues.

Workaround

1. Take all nodes out of rotation. This avoids concurrency.
2. Shut down all but one node.
3. Perform the lock-and-reindex on the remaining node.
4. Start the other nodes and wait for index replication to complete.
5. Put all nodes back into rotation.