Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-19937

Post Bitbucket 9.3 locks are not able to be acquired in a cluster where one of the nodes is shutdown

      Issue Summary

      Starting with Bitbucket 9.3, when managing a cluster, you may encounter issues if you attempt to manually lock the cluster or perform backups while one of the nodes is shut down or in a stopped state. In such cases, the lock might not be acquired successfully. Even if the lock is acquired, you may face difficulties unlocking the instance later, which requires a instance restart.

      Steps to Reproduce:

      1. Setup a Bitbucket Data Center Cluster in version 9.3 or higher.
      2. Shutdown one of your nodes.
      3. Manually lock your instance with below endpoint
        curl -s \
        	 -u ${BITBUCKET_BACKUP_USER}:${BITBUCKET_BACKUP_PASS} \
        	 -X POST \
        	 -H "Content-type: application/json" \
             "${BITBUCKET_URL}/mvc/maintenance/lock"
        
      1. Check logs and see if lock was able to be acquired.
      2. Try to unlock you instance with no success if lock was acquired.

      Expected Results:

      Lock is able to be acquired and instance unlock successful.

      Actual Results:

      Below error is thrown in the logs when lock is ran for the first time and when trying to unlock using the token found in the logs:

      2025-02-07 03:00:07,720 DEBUG [http-nio-7990-exec-4] admin @RMTUB2x799x390x0 10.224.217.110,10.224.217.84 "POST /mvc/maintenance/lock HTTP/1.0" c.a.s.i.a.DefaultAuthenticationService username_test: Authenticator 'com.atlassian.bitbucket.server.bitbucket-authentication:crowdHttpAuthHandler' authenticated admin (1)
      2025-02-13 13:19:01,735 ERROR [http-nio-7990-exec-7] admin @RMTUB2x799x390x0 10.224.217.110,10.224.217.84 "POST /mvc/maintenance/lock HTTP/1.1" c.a.s.i.w.HandlerExceptionResolverImpl ExecutorService[bitbucket.core] is shutdown! In order to create a new ExecutorService with name 'bitbucket.core', you need to destroy current ExecutorService first!
      	at com.atlassian.stash.internal.maintenance.ClusterMaintenanceLock.lock(ClusterMaintenanceLock.java:56)
      	at com.atlassian.stash.internal.maintenance.DefaultMaintenanceService.lock(DefaultMaintenanceService.java:172)
      	at jdk.proxy3/jdk.proxy3.$Proxy329.lock(Unknown Source)
      	at com.atlassian.stash.internal.web.maintenance.MaintenanceController.lock(MaintenanceController.java:185)
      

      Workaround:

      Restart instance and run locks with all nodes up.

          Form Name

            [BSERV-19937] Post Bitbucket 9.3 locks are not able to be acquired in a cluster where one of the nodes is shutdown

            There are no comments yet on this issue.

              e74118b4e28d David Jansons
              13a66df57723 Victor Menes
              Affected customers:
              2 This affects my team
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: