Mirror's unexpected removal may overload the upstream cluster

Issue Summary

Unexpected removal of a Smart Mirror may result with overloading all nodes of the upstream cluster

Steps to Reproduce

1. Copy your Bitbucket Home and the database from Production to Staging
   The copied database references a Production Mirror
2. Start Bitbucket Data Center in Staging (for this exercise, a single node is sufficient)
3. Login to Bitbucket Data Center in Staging and see the list of existing Mirrors
   The Production Mirror should be listed
4. Using the web interface, Remove the Production Mirror from Staging

Expected Results

1. The Mirror should be unreferenced from Staging, but not removed from Production
2. If the Mirror is unexpectedly removed from Production, the Production cluster should not overload itself trying to contact the removed Mirror

Actual Results

1. Removing the Production Mirror from Staging sends a REMOVE message to the Mirror
2. The Mirror removes itself from its upstream cluster (which is the Production cluster!)
3. For the Production cluster, the Mirror's removal is unexpected.
   All the Production nodes attempt to contact the removed Mirror. These attempts fail, and are repeated again and again, to the point of overloading the entire Production cluster (all nodes), making it unresponsive.

The below message is logged in the Mirror's atlassian-bitbucket-access.log file:

INFO  [httpclient-callbacks:thread-985] @1IP3TKx1221x58x0 ##.##.##.##,##.##.##.## "HEAD /status HTTP/1.1" c.a.b.i.m.m.DefaultUpstreamService Setting upstream state for @@@@@@@@-@@@@-@@@@-@@@@-@@@@@@@@@@@@ to REMOVED

Workaround

1. Using the firewall, or each node's /etc/hosts file, ensure a complete isolation of the Staging environment from the Production environment.
2. Before starting the Staging cluster, modify the database, to remove the reference to the existing Production Mirror:

   DELETE FROM plugin_setting WHERE key_name LIKE 'acnct.bitbucket.mirror.%'
   

   NOTE: While this seems to be working as expected, it is not a supported approach, and it may only be used in an isolated Staging environment.