Issue Summary

Backing up Bitbucket DC with repositories on remote mesh nodes using the zero downtime strategy can sometimes cause inconsistencies in some repositories on those mesh nodes after they are restored.

This is reproducible on Data Center: yes

Steps to Reproduce

• Install a Bitbucket main node with an EBS volume for the Home directory, an RDS, and three mesh nodes each with an EBS volume for their home directories.
• Set up the zero downtime script.
• Run the zero downtime script while push operations are in progress.
• Restore Bitbucket from the backup.
• Check the consistency of the repository that was being pushed during the operation.

Expected Results

Repository should be consistent after the restoration.

Actual Results

Repository(s) is inconsistent after the restoration.

Workaround

• By having a replication factor of 3, you can shut down one of three Mesh node while keeping Bitbucket available for normal operations.
• Before taking a snapshot, shut down the Mesh node whose snapshot to be taken, and start it up again after the snapshot is complete.
• When launching Bitbucket after a backup restoration of mesh nodes, use the disaster.recovery flag in bitbucket.properties file to prompt Bitbucket to instruct Mesh to resolve any inconsistencies.