Details
High Description
Problem
Bamboo will not trigger an Elastic Agent shutdown when the Idle agent shutdown delay time expires and the Elastic Agent had failed to send the build results' content back to the server at any past stages.
Environment
Bamboo 8.2, 9.0, 9.1
Steps to Reproduce
- Set a very short Elastic Agent Idle agent shutdown delay time
- Create a plan that would create a large build-results.xml file on the build-dir
- Launch an Elastic Agent on a very small filesystem; we are looking to simulate a disk full error - Use a small EBS volume to have a quick environment
- Run the Plan so it will occupy 100% of the disk and generate a "No space left on device" error and a stack trace
- The Elastic Agent will try to transmit the build-results.xml file and will fail, but it will continue working and end up changing the build context back to null, enabling further Jobs to be queued and built
- From that point and onwards, the Bamboo Server will not send a shutdown command back to the Elastic Agent even if the Idle agent shutdown delay time expires as it would consider the Agent as busy due to the missing build results
Expected Results
Bamboo should be able to process the build results and send them to the Server regardless of a disk issue; or, the Server should consider the idleness of the Agent, even if it failed to send the build results back to it
Actual Results
Bamboo will not trigger an Elastic Agent shutdown when the Idle agent shutdown delay time expires and the Elastic Agent had failed to send the build results' content back to the server at any past stages.
Other than the technical impact, customers expect Agents to be shut down when idle due to financial constraints. For some customers, having EC2 instances hanging around without a real purpose is considered a critical issue.
Workaround
Prevent Bamboo from using 100% of the disk either by extending it or modifying the build results or implementing a Plan cleanup policy.