Bamboo always prioritizes elastic images that have the biggest number of builds relying on them when starting up new elastic agents

Issue Summary

There's a problem in the mechanism that calculates the number of elastic agents that are needed to build the items that are in the queue. Bamboo always prioritizes elastic images that have the biggest number of builds relying on them.

If you always have a large number of builds on the queue that depend on the same image they will constantly get prioritized. For example, let's say you have 50 builds in the queue and trigger 20 deployment environments (no matter the order that they are in the queue). If those 50 builds depend on the same elastic image, they will be prioritized over the 20 deployment environments. Bamboo will start more elastic agents to handle those 50 builds than elastic agents to deal with the 20 deployment environments. If this scenario keeps on repeating throughout the day then those builds will constantly get prioritized over the deployment environments.

This won't stop the deployment environments from running but it will take considerably more time for them to run since they will have less elastic agents available to build them.

This is reproducible on Data Center:

Steps to Reproduce

1. Set up a similar automatic elastic instance management configuration to allow for multiple elastic agents to run in Bamboo:
   
2. Create 3 plans in Bamboo that do something simple (e.g. echo test).
3. Enable concurrent builds (e.g. 50).
4. Create a deployment project called ABC with about 10 deployment environments. It's up to you if you want to create a separate plan for this deployment project. You can reuse one of the plans you created in the previous step.
5. Set up two identical elastic images based off of the Ubuntu stock image and disable all other images:
   • Ubuntu 1 dedicate to building your deployment project called ABC.
   • Ubuntu 2 leave with no dedication so it can pick up the rest of your builds.
6. Trigger several builds from the plans you created previously (e.g. +/- 20 builds in total).
7. Once you have about 20 builds on the queue already trigger the 10 deployment environments from ABC.

Expected Results

Bamboo will start up a mix of several elastic agents based on images Ubuntu 1 and Ubuntu 2 to handle the builds and deployment environments on the queue.

Actual Results

First Bamboo will start about 9 elastic agents (Ubuntu 2) to handle your plans and 1 elastic agent (Ubuntu 1) to handle ABC. The number of elastic images it can start up at once is controlled by your automatic elastic instance management configuration. After about a minute Bamboo will start 10 more elastic agents (Ubuntu 2) to handle your plans but nothing else for your deployment environments.

Here's where the problem is - there were more builds in the queue from other plans than from ABC so Bamboo will keep on creating more and more elastic agents to handle builds and will never start more than 1 elastic agent to handle ABC. All of your 10 deployment environments from ABC will end up waiting for a single elastic agent to become online.

Workaround

If your builds are running on a schedule (e.g. every day 12 PM) you can configure an instance schedule to start up elastic agents for the images that are affected by this problem just before the builds get triggered.

For example, if your builds get triggered every day at 12 PM you can create an instance schedule to start whatever number of elastic agents are needed for those builds a few minutes (e.g. 11:50 AM) before they get triggered. This will help ensure that your builds will have enough elastic agents to run on by the time they get queued.

Notes

This issue doesn't happen all the time though. It fluctuates based on the volume of the queue. It depends on how many builds are in the queue when your deployment environments got triggered (following the example above). When there's little to no activity in the queue your 10 deployment environments will end up with several elastic agents named Ubuntu 1 to build them and not only one.