-
Bug
-
Resolution: Cannot Reproduce
-
High
-
None
-
4.14.9, 6.1.0, 7.2.4
-
11
-
Severity 2 - Major
-
20
-
Issue Summary
Unexpected removal of a Smart Mirror may result with overloading all nodes of the upstream cluster
Steps to Reproduce
- Copy your Bitbucket Home and the database from Production to Staging
The copied database references a Production Mirror - Start Bitbucket Data Center in Staging (for this exercise, a single node is sufficient)
- Login to Bitbucket Data Center in Staging and see the list of existing Mirrors
The Production Mirror should be listed - Using the web interface, Remove the Production Mirror from Staging
Expected Results
- The Mirror should be unreferenced from Staging, but not removed from Production
- If the Mirror is unexpectedly removed from Production, the Production cluster should not overload itself trying to contact the removed Mirror
Actual Results
- Removing the Production Mirror from Staging sends a REMOVE message to the Mirror
- The Mirror removes itself from its upstream cluster (which is the Production cluster!)
- For the Production cluster, the Mirror's removal is unexpected.
All the Production nodes attempt to contact the removed Mirror. These attempts fail, and are repeated again and again, to the point of overloading the entire Production cluster (all nodes), making it unresponsive.
The below message is logged in the Mirror's atlassian-bitbucket-access.log file:
INFO [httpclient-callbacks:thread-985] @1IP3TKx1221x58x0 ##.##.##.##,##.##.##.## "HEAD /status HTTP/1.1" c.a.b.i.m.m.DefaultUpstreamService Setting upstream state for @@@@@@@@-@@@@-@@@@-@@@@-@@@@@@@@@@@@ to REMOVED
Workaround
- Using the firewall, or each node's /etc/hosts file, ensure a complete isolation of the Staging environment from the Production environment.
- Before starting the Staging cluster, modify the database, to remove the reference to the existing Production Mirror:
DELETE FROM plugin_setting WHERE key_name LIKE 'acnct.bitbucket.mirror.%'
NOTE: While this seems to be working as expected, it is not a supported approach, and it may only be used in an isolated Staging environment.
- relates to
-
BSERV-13130 Deleting a mirror on a copy of bitbucket also removes the mirror from the original
- Closed
- is blocked by
-
PS-29744 Loading...
- mentioned in
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...