Unexpected removal of a Smart Mirror may result with overloading all nodes of the upstream cluster
- Copy your Bitbucket Home and the database from Production to Staging
The copied database references a Production Mirror
- Start Bitbucket Data Center in Staging (for this exercise, a single node is sufficient)
- Login to Bitbucket Data Center in Staging and see the list of existing Mirrors
The Production Mirror should be listed
- Using the web interface, Remove the Production Mirror from Staging
- The Mirror should be unreferenced from Staging, but not removed from Production
- If the Mirror is unexpectedly removed from Production, the Production cluster should not overload itself trying to contact the removed Mirror
- Removing the Production Mirror from Staging sends a REMOVE message to the Mirror
- The Mirror removes itself from its upstream cluster (which is the Production cluster!)
- For the Production cluster, the Mirror's removal is unexpected.
All the Production nodes attempt to contact the removed Mirror. These attempts fail, and are repeated again and again, to the point of overloading the entire Production cluster (all nodes), making it unresponsive.
The below message is logged in the Mirror's atlassian-bitbucket-access.log file:
- Using the firewall, or each node's /etc/hosts file, ensure a complete isolation of the Staging environment from the Production environment.
- Before starting the Staging cluster, modify the database, to remove the reference to the existing Production Mirror:
NOTE: While this seems to be working as expected, it is not a supported approach, and it may only be used in an isolated Staging environment.