Summary

The greater the number of failed or skipped tests, the longer it takes for Bamboo to process them. By default, Bamboo operates with a single thread, which I will refer to as the "expensive" thread in this report, responsible for handling the processing of build and deployment results. If Bamboo spends an extended period processing tests from a single build, it can lead to the expensive thread being occupied for several minutes to hours while dealing with these tests. Consequently, severe performance issues may arise since Bamboo will be unable to process the results of any other builds or deployments until the expensive thread becomes available again. This can cause builds to be canceled and removed from the queue, with the message "Build had to be canceled: it was marked as queued but was not present in the queue for at least the past 720 seconds". Furthermore, this situation could potentially trigger out-of-memory errors due to the need to store these tests in memory while the results are being processed.

Steps to Reproduce

1. Create a plan with more than 4000 test results.
2. Make sure all of them are skipped or fail.
3. Check how long it takes to finish the build after the below message is shown:

   bamboo-stdout.2017-12-21.log:2017-12-22 11:24:03,110 INFO [20-BAM::Mehlinger::Agent:pool-36-thread-1] [PlanStatePersisterImpl] PLAN-KEY-JOB1-1: Generating build results summary...
   

Expected Results

Processing build results shouldn't take that long since the build work was already done.

Actual Results

It might take hours to finish (in this example 7.7 hours):

bamboo-stdout.2017-12-21.log:2017-12-22 11:24:03,110 INFO [20-BAM::Mehlinger::Agent:pool-36-thread-1] [PlanStatePersisterImpl] PLAN-KEY-JOB1-1: Generating build results summary...
bamboo-stdout.2017-12-21.log:2017-12-22 11:24:03,266 WARN [20-BAM::Mehlinger::Agent:pool-36-thread-1] [TestCaseResultHibernateDao] Unbounded test query detected from getTestsForBuildResultByState, capping at 1000
bamboo-stdout.2017-12-21.log:2017-12-22 15:16:28,736 WARN [20-BAM::Mehlinger::Agent:pool-36-thread-1] [TestCaseResultHibernateDao] Unbounded test query detected from getTestsForBuildResultByState, capping at 1000
bamboo-stdout.2017-12-21.log:2017-12-22 19:11:06,320 INFO [20-BAM::Mehlinger::Agent:pool-36-thread-1] [TestsManagerImpl] Test migration to BuildResultsSummary complete for PLAN-KEY-JOB1-1, 674 test classes, 4574 test cases, time elapsed: 7.784 h

Workaround

We recommend using one (or more) of the following solutions to workaround the issue:

1. Consider reducing the number of tests in the plan(s) causing problems.
2. Split your job(s) with the highest number of tests into many smaller ones. This can be achieved using Maven test groups, for example. Although this may extend the overall duration of the build, it will distribute the processing of test cases over a longer time period and reduce the need for many resources. This will lower the chances of causing performance issues.
3. Stop parsing test results for the plan(s) causing problems. For instance, convert the test results to an HTML report during the build and return it to Bamboo as an artifact.
   • This approach will result in a loss of capability to view the test results within the build result summary page and the ability to quarantine tests.
4. If Bamboo isn't running out of memory, but it's taking a significant amount of time to process tests for a specific plan, and you're unable to further reduce the number of tests, consider increasing the number of available expensive threads in Bamboo. This will ensure that even if Bamboo takes an extended time processing tests for one build, there will be additional threads available to handle the results from your other builds and deployments.

   -Dbamboo.max.concurrent.expensive.messages=2
   

   Please refer to the "Configuring your system properties" page on how to add this property to your Bamboo instance.

   Important

   Exercise caution when increasing the number of expensive threads. Allowing Bamboo to process multiple build and deployment results simultaneously by enabling multiple expensive threads can potentially lead to the processing of builds with a high volume of test cases concurrently, further deteriorating the application performance. Before adjusting the number of expensive threads, we recommend evaluating your application's performance to determine if there's a need to increase the heap size (see Tip 1 below).

5. If Bamboo is running out of memory already, or if it began happening after completing step 3, it means the application lacks the necessary memory to handle the tests from the affected plan(s). To resolve this issue, you'll have to increase the heap size allocated to your Bamboo instance. Detailed instructions for doing so can be found on the "OutOfMemory Errors in Bamboo server or in remote agent" page.