Shutting down Bitbucket while requests are in flight can leave a stale Liquibase changelog lock in the database

XMLWordPrintable

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Medium
    • None
    • Affects Version/s: 7.21.0
    • Component/s: Spring, StartUp
    • None
    • 3
    • Severity 3 - Minor
    • 45

      Issue Summary

      During a shutdown of Bitbucket, a race condition can occur where a thread that was already processing work (such as handling a git hook callback, a Hazelcast cluster operation, or any other in-flight request) remains active after Spring has already destroyed its managed beans. This thread can trigger Spring to recreate certain beans that have already been shut down, including the Liquibase bean responsible for database schema management. If the JVM exits before this bean has finished initialising and releases the changelog lock, the databasechangeloglock table row is left in a locked state. On the next startup, Bitbucket attempts to acquire this lock and waits until the configured timeout expires before failing, preventing the node from starting.

      In the observed case, the in-flight thread was handling a git hook callback from a long-running SSH git fetch or git clone operation that had started before the shutdown was initiated.

      Steps to Reproduce

      Note: The race condition window is very narrow and difficult to reproduce naturally. The following steps describe the conditions under which it can occur.

      1. Start a long-running SSH git fetch or git clone operation against a Bitbucket repository
      2. While the operation is still in progress, initiate a graceful shutdown of Bitbucket
      3. If the shutdown and the git hook callback overlap in the right window, the race condition occurs and the databasechangeloglock row is left locked in the database
      4. Attempt to start Bitbucket again

      Expected Results

      Bitbucket starts up successfully. The databasechangeloglock is not locked.

      Actual Results

      Bitbucket hangs on startup for the duration of the configured db.schema.lock.maxWait (default 30 minutes) then fails with the following error in the atlassian-bitbucket.log file:

      ERROR [spring-startup] JohnsonDispatcherServlet SpringMVC dispatcher [springMvc] could not be started
      org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name 'crowdAliasDao': ...
      Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'bootstrapLiquibase' ...
          Invocation of init method failed; nested exception is liquibase.exception.LockException:
          Could not acquire change log lock. Currently locked by <hostname> since <timestamp>
      	at liquibase.lockservice.StandardLockService.waitForLock(StandardLockService.java:270)
      	at liquibase.integration.spring.SpringLiquibase.performUpdate(SpringLiquibase.java:314)
      	at com.atlassian.stash.internal.liquibase.database.ExtendedSpringLiquibase.afterPropertiesSet(ExtendedSpringLiquibase.java:63)
      

      The databasechangeloglock table will show a locked row with the hostname of the affected node and a timestamp matching the previous shutdown:

      SELECT * FROM databasechangeloglock;
      
       id | locked |       lockgranted        |         lockedby
      ----+--------+--------------------------+--------------------------
        1 | t      | 2026-05-07 10:06:08      | <hostname>
      

      Workaround

      The stale lock can be cleared manually by running the following SQL against the Bitbucket database:

      UPDATE databasechangeloglock SET LOCKED=false, LOCKGRANTED=null, LOCKEDBY=null WHERE ID=1;
      

      After running this, restart Bitbucket and it should start successfully.

      See also: https://support.atlassian.com/bitbucket-data-center/kb/bitbucket-server-does-not-start-could-not-acquire-change-log-lock/

              Assignee:
              Unassigned
              Reporter:
              Josh Aguilar
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Created:
                Updated: