If the DIY Backup fails, the Bitbucket instance becomes unusable and the status is stuck in the MAINTENANCE mode.

Steps to reproduce:

1. Set up Bitbucket DIY Backup as per usual.
2. Purposely make the backup fail; bitbucket.diy-backup.vars.sh:

   CURL_OPTIONS="-L -s -f"
   BITBUCKET_VERBOSE_BACKUP=true
   
   INSTANCE_NAME=bitbucket
   BITBUCKET_URL=http://localhost:7990
   
   BITBUCKET_HOME=/var/atlassian/application-data/bitbucket
   BITBUCKET_UID=bitbucket
   BITBUCKET_GID=bitbucket
   
   BACKUP_DISK_TYPE=rsync
   BACKUP_DATABASE_TYPE=postgresql
   BACKUP_ARCHIVE_TYPE=tar
   
   
   # The admin user and password for the Bitbucket instance
   BITBUCKET_BACKUP_USER=admin
   BITBUCKET_BACKUP_PASS=admin
   
   BITBUCKET_BACKUP_EXCLUDE_REPOS=()
   
   # PostgreSQL configuration
   BITBUCKET_DB=bb1
   POSTGRES_HOST=172.17.0.2
   POSTGRES_USERNAME=postgres
   export PGPASSWORD=postgres
   POSTGRES_PORT=5432
   
   # The path to working folder for the backup
   BITBUCKET_BACKUP_ROOT=/tmp/bitbucket-backup
   BITBUCKET_BACKUP_DB=${BITBUCKET_BACKUP_ROOT}/database
   BITBUCKET_BACKUP_HOME=${BITBUCKET_BACKUP_ROOT}/home
   

   BITBUCKET_DB=bb1 - the database does not exist.

3. Run the backup:

   [common.sh:17 -> source][be8cd16cda00] DEBUG: Using vars file: './bitbucket.diy-backup.vars.sh'
   [bitbucket.diy-backup.sh:0 -> main][be8cd16cda00]  INFO: Preparing for backup
   [common.sh:123 -> lock_bitbucket][be8cd16cda00] DEBUG: Running curl -L -s -f -u ******:****** http://localhost:7990/mvc/maintenance/lock
   [common.sh:123 -> lock_bitbucket][be8cd16cda00]  INFO: Bitbucket has been locked for maintenance.  It can be unlocked with:
   [common.sh:123 -> lock_bitbucket][be8cd16cda00]  INFO:     curl -u ... -X DELETE -H 'Content-type:application/json' 'http://localhost:7990/mvc/maintenance/lock?token=e3d8887ce940d423ebb483bd975b1430343c6efc'
   [common.sh:124 -> backup_start][be8cd16cda00] DEBUG: Running curl -L -s -f -u ******:****** http://localhost:7990/mvc/admin/backups?external=true
   [common.sh:124 -> backup_start][be8cd16cda00]  INFO: Bitbucket server is now preparing for backup. If the backup task is cancelled, Bitbucket Server should be notified that backup was terminated by executing the following command:
   [common.sh:124 -> backup_start][be8cd16cda00]  INFO:     curl -u ... -X POST -H 'Content-type:application/json' 'http://localhost:7990/mvc/maintenance?token=c69169060759603ea491e55bfce2d184391d1afb'
   [common.sh:124 -> backup_start][be8cd16cda00]  INFO: This will also terminate the backup process in Bitbucket Server. Note that this will not unlock Bitbucket Server from maintenance mode.
   [bitbucket.diy-backup.sh:127 -> run_in_bg][be8cd16cda00] DEBUG: Started Elasticsearch backup (PID=809)
   [common.sh:129 -> backup_wait][be8cd16cda00]  INFO: Waiting for Bitbucket to become ready to be backed up
   [bitbucket.diy-backup.sh:0 -> main][be8cd16cda00]  INFO: Backing up the database and filesystem in parallel
   [bitbucket.diy-backup.sh:132 -> run_in_bg][be8cd16cda00] DEBUG: Started Database backup (PID=834)
   [bitbucket.diy-backup.sh:133 -> run_in_bg][be8cd16cda00] DEBUG: Started Disk backup (PID=840)
   [database-postgresql.sh:68 -> backup_db][be8cd16cda00] DEBUG: Running pg_dump -U postgres -h 172.17.0.2 --port=5432 -j 5 -Fd -d bb1 -f /tmp/bitbucket-backup/database
   [bitbucket.diy-backup.sh:136 -> wait_for_bg_jobs][be8cd16cda00] DEBUG: Waiting for Disk backup (PID=840)
   [bitbucket.diy-backup.sh:136 -> wait_for_bg_jobs][be8cd16cda00] DEBUG: Disk backup finished successfully (PID=840)
   [common.sh:85 -> update_backup_progress][be8cd16cda00] DEBUG: Running curl -L -s -f -u ******:****** http://localhost:7990/mvc/admin/backups/progress/client?token=e3d8887ce940d423ebb483bd975b1430343c6efc&percentage=50
   pg_dump: error: connection to server at "172.17.0.2", port 5432 failed: FATAL:  database "bb1" does not exist
   [bitbucket.diy-backup.sh:136 -> wait_for_bg_jobs][be8cd16cda00] DEBUG: Waiting for Database backup (PID=834)
   [bitbucket.diy-backup.sh:136 -> wait_for_bg_jobs][be8cd16cda00] DEBUG: Waiting for Elasticsearch backup (PID=809)
   [bitbucket.diy-backup.sh:136 -> wait_for_bg_jobs][be8cd16cda00] DEBUG: Elasticsearch backup finished successfully (PID=809)
   [common.sh:85 -> update_backup_progress][be8cd16cda00] DEBUG: Running curl -L -s -f -u ******:****** http://localhost:7990/mvc/admin/backups/progress/client?token=e3d8887ce940d423ebb483bd975b1430343c6efc&percentage=50
   [bitbucket.diy-backup.sh:136 -> wait_for_bg_jobs][be8cd16cda00] ERROR: Database backup failed with status 1 (PID=809)
   [common.sh:138 -> unlock_bitbucket][be8cd16cda00] DEBUG: Running curl -L -s -f -u ******:****** http://localhost:7990/mvc/maintenance/lock?token=e3d8887ce940d423ebb483bd975b1430343c6efc
   [bitbucket.diy-backup.sh:139 -> cleanup_incomplete_backup][be8cd16cda00] DEBUG: Cleaning up after failed backup
   [bitbucket.diy-backup.sh:0 -> main][be8cd16cda00] ERROR: Backing up Bitbucket failed
   [common.sh:1 -> run_cleanup][be8cd16cda00] DEBUG: Running cleanup jobs...
   

   Database backup will fail:

   pg_dump: error: connection to server at "172.17.0.2", port 5432 failed: FATAL:  database "bb1" does not exist
   

4. Application log; atlassian-bitbucket.log:

   2024-05-20 00:46:26,636 INFO  [hz.hazelcast.cached.thread-4]  c.a.s.i.m.DefaultMaintenanceService The system has been locked for maintenance. It may be unlocked with token: 9a346866f47e8baa310b3a19e5ed6c3b0ff12af1
   2024-05-20 00:46:26,652 DEBUG [hz.hazelcast.async.thread-2]  c.a.s.i.m.ClusterMaintenanceLock$FailureTrackingCallback Success executing on Member [172.17.0.3]:5701 - 16dce5e3-9b0f-4e44-a7ec-b9d7f895a5cf this. Result: null
   2024-05-20 00:46:26,660 DEBUG [hz.hazelcast.async.thread-2]  c.a.s.i.m.ClusterMaintenanceLock$FailureTrackingCallback Execution complete. Successful - true
   2024-05-20 00:46:56,870 INFO  [http-nio-7990-exec-8] @1B2YTOLx46x12x0 lbo7jb 192.168.65.1 "GET /projects HTTP/1.1" c.a.johnson.filters.JohnsonFilter The application is still starting up, or there are errors. Redirecting request from '/projects' to '/unavailable'
   2024-05-20 00:46:58,295 INFO  [hz.hazelcast.cached.thread-4]  c.a.s.i.m.DefaultMaintenanceLock Maintenance has been completed. The system lock has been released
   2024-05-20 00:46:58,301 DEBUG [hz.hazelcast.async.thread-1]  c.a.s.i.m.ClusterMaintenanceLock$FailureTrackingCallback Success executing on Member [172.17.0.3]:5701 - 16dce5e3-9b0f-4e44-a7ec-b9d7f895a5cf this. Result: null
   2024-05-20 00:46:58,304 DEBUG [hz.hazelcast.async.thread-1]  c.a.s.i.m.ClusterMaintenanceLock$FailureTrackingCallback Execution complete. Successful - true
   

Current behaviour:

1. The DIY backup will fail but Bitbucket status will stuck in the MAINTENANCE mode.

   $: curl http://localhost:7990/status
   {"state":"MAINTENANCE"}
   

2. Unable to load any Bitbucket page.

Expected behaviour:

1. The DIY backup will fail but Bitbucket status will not be in the MAINTENANCE mode.
2. Bitbucket should be in a usable state.

Workaround

1. Restart the Bitbucket will solve the status and page loading.