Uploaded image for project: 'Bitbucket Server'
  1. Bitbucket Server
  2. BSERV-11510

Mirror's unexpected removal may overload the upstream cluster

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • High
    • Resolution: Cannot Reproduce
    • 4.14.9, 6.1.0, 7.2.4
    • None
    • Smart Mirroring

    Description

      Issue Summary

      Unexpected removal of a Smart Mirror may result with overloading all nodes of the upstream cluster

      Steps to Reproduce

      1. Copy your Bitbucket Home and the database from Production to Staging
        The copied database references a Production Mirror
      2. Start Bitbucket Data Center in Staging (for this exercise, a single node is sufficient)
      3. Login to Bitbucket Data Center in Staging and see the list of existing Mirrors
        The Production Mirror should be listed
      4. Using the web interface, Remove the Production Mirror from Staging

      Expected Results

      1. The Mirror should be unreferenced from Staging, but not removed from Production
      2. If the Mirror is unexpectedly removed from Production, the Production cluster should not overload itself trying to contact the removed Mirror

      Actual Results

      1. Removing the Production Mirror from Staging sends a REMOVE message to the Mirror
      2. The Mirror removes itself from its upstream cluster (which is the Production cluster!)
      3. For the Production cluster, the Mirror's removal is unexpected.
        All the Production nodes attempt to contact the removed Mirror. These attempts fail, and are repeated again and again, to the point of overloading the entire Production cluster (all nodes), making it unresponsive.

      The below message is logged in the Mirror's atlassian-bitbucket-access.log file:

      INFO  [httpclient-callbacks:thread-985] @1IP3TKx1221x58x0 ##.##.##.##,##.##.##.## "HEAD /status HTTP/1.1" c.a.b.i.m.m.DefaultUpstreamService Setting upstream state for @@@@@@@@-@@@@-@@@@-@@@@-@@@@@@@@@@@@ to REMOVED
      

      Workaround

      1. Using the firewall, or each node's /etc/hosts file, ensure a complete isolation of the Staging environment from the Production environment.
      2. Before starting the Staging cluster, modify the database, to remove the reference to the existing Production Mirror:
        DELETE FROM plugin_setting WHERE key_name LIKE 'acnct.bitbucket.mirror.%'
        

        NOTE: While this seems to be working as expected, it is not a supported approach, and it may only be used in an isolated Staging environment.

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              msuchecki Marek S.
              Votes:
              2 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Backbone Issue Sync