Bamboo fails to startup due to invalid branch keys in the BUILD table

Summary

Bamboo fails to startup after an application restart due to invalid branch keys in the BUILD table. The following errors show up inside the application log file:

2017-07-28 13:10:20,862 INFO [localhost-startStop-1] [AbstractUpgradeManager] -------------------------------------------------------------------------
2017-07-28 13:10:20,862 INFO [localhost-startStop-1] [AbstractUpgradeManager] 4410 : Make sure that all branch keys start with their master's chain key
2017-07-28 13:10:20,862 INFO [localhost-startStop-1] [AbstractUpgradeManager] -------------------------------------------------------------------------
2017-07-28 13:10:20,869 INFO [localhost-startStop-1] [AbstractUpgradeManager] Completed task 4410 with errors.
...
2017-07-28 13:10:21,012 FATAL [localhost-startStop-1] [DefaultBootstrapManager] Pre-bootstrap upgrade tasks failed
com.atlassian.upgrade.UpgradeException: Pre-bootstrap validation tests failed: [Task for build 4410 failed:, com.atlassian.bamboo.upgrade.tasks.validation.BranchKeyStartsWithMastersChainKey: There are invalid branch keys in the BUILD table: [GIT-HUB2]]
        at com.atlassian.bamboo.setup.DefaultBootstrapManager.runValidationTasks(DefaultBootstrapManager.java:482)
        at com.atlassian.bamboo.setup.DefaultBootstrapManager.performPersistenceUpgrade(DefaultBootstrapManager.java:340)
        at com.atlassian.config.bootstrap.DefaultAtlassianBootstrapManager.init(DefaultAtlassianBootstrapManager.java:77)
        at com.atlassian.bamboo.setup.BootstrapLoaderListener.contextInitialized(BootstrapLoaderListener.java:118)
        at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4842)
        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5303)
        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:147)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1407)
        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1397)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Steps to Reproduce

1. Create a plan with a few plan branches.
2. Delete some of the plan branches you just created. Due to performance reasons, they're not removed immediately.
3. Go to Bamboo administration >> Overview >> Plans >> Move plans.
4. Move the plan you just created to a new project.
5. Restart Bamboo.

The problem happens because plan branches are not immediately removed once they're set to be deleted. Once you move a plan to a new project or just rename it, the operation doesn't move plan branches marked for deletion. If then Bamboo is restarted before the plan branches get removed, that validation task will fail because it will detect orphaned plan branches.

Expected Results

Bamboo should be able to start up without errors.

Actual Results

Bamboo fails to startup because the validation task 4410 completed with errors.

Notes

Bamboo version 5.11.3 or lower

If you're running an older version of Bamboo (e.g. 5.9.7, 5.10.3, 5.11.3), then it's more likely you are affected by BAM-15524 and you should upgrade to version 5.12.0.2 or higher to fix that problem, and the Workaround here may not apply to you.

Bamboo version 5.12.0.2 or higher

If you are running Bamboo 5.12.0.2 or higher, it's more likely that you're affected by the problem described here and therefore the Workaround applies to you.

Workaround

1. Stop Bamboo.
2. Open up the <BAMBOO_INSTALL>/atlassian-bamboo/WEB-INF/classes/upgrades.xml file.
3. Find the following entry and comment it out:
   From

   <upgrade build="4410" build-min="4410" build-max="999999"
                class="com.atlassian.bamboo.upgrade.tasks.validation.BranchKeyStartsWithMastersChainKey"/>
   

   To

   <!-- <upgrade build="4410" build-min="4410" build-max="999999"
                class="com.atlassian.bamboo.upgrade.tasks.validation.BranchKeyStartsWithMastersChainKey"/> -->
   

4. Save the file and restart Bamboo.
5. Once that's been done, Bamboo will startup and the removal job will kick in and delete the plan branches that should've been removed already.
   • You can check whether the plan branches were removed running the following SQL statement inside the Bamboo database:

     SELECT *
       FROM build
      WHERE build_type = 'CHAIN_BRANCH'
        AND full_key = '<PLAN_BRANCH_KEY>';
     

     Replace <PLAN_BRANCH_KEY> with one of the invalid branch keys mentioned in the error.

6. Confirm the plan branches don't exist anymore i.e. removal job expired them.
7. Revert the changes made to the upgrades.xml file and restart Bamboo one more time.