Uploaded image for project: 'Bamboo Data Center'
  1. Bamboo Data Center
  2. BAM-25838

In a Bamboo instance with a large number of remote agents and a large number of capabilities, we could see performance issues while adding jobs to the queue

XMLWordPrintable

    • 2
    • Our product teams collect and evaluate feedback from a number of different sources. To learn more about how we use customer feedback in the planning process, check out our new feature policy.

      Issue Summary

      This is reproducible on Data Center: Yes

      In a Bamboo instance with a large number of remote agents and a large number of capabilities, we could see performance issues while adding jobs to the queue.

      This can result in long queue locking that means agents cannot efficiently take builds from the build queue:

      2024-07-02 14:28:47,432 ERROR [ActiveMQ Session Task-38382] [BambooLogUtils] Bamboo operation time: takeBuildContext took 57831 ms
      

      Steps to Reproduce

      1. Create more than ~250 remote agents each with more than 300 remote capabilities 
      2. Create a plan with 150 jobs
      3. run the build plan and monitor the logs

      While calculating the executable agent for each job that has been queued it also tries to combine the shared capabilities. Even though sharedCapabilitySet might have no shared-capabilities defined on the instance the method to combine is executed and checked. 

      We are requesting that the code be enhanced by checking whether any shared capabilities are defined in the set and executing the combination of shared and remote capabilities.

      The below improvements were seen (with custom timing logging added) while testing after improving the code:

      Shared capability:
      2024-07-18 08:50:29,926 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-257] [AbstractQueueOfExecutables] Time to calculateAgents for PROJ-PLAN1-JOB100-34 36 ms
      2024-07-18 08:50:29,932 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-257] [BuildQueueManagerImpl] Sent ExecutableQueueUpdate: addToQueue, agents known to be affected: []
      2024-07-18 08:50:29,932 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-257] [BuildQueueManagerImpl] Time to addtoQueue for PROJ-PLAN1-JOB100-34 5113 ms
      
      No shared capability:
      2024-07-18 08:51:45,497 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-278] [AbstractQueueOfExecutables] Time to calculateAgents for PROJ-PLAN1-JOB100-35 7 ms
      2024-07-18 08:51:45,498 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-278] [BuildQueueManagerImpl] Sent ExecutableQueueUpdate: addToQueue, agents known to be affected: []
      2024-07-18 08:51:45,498 INFO [AtlassianEvent::0-BAM::EVENTS:pool-2-thread-278] [BuildQueueManagerImpl] Time to addtoQueue for PROJ-PLAN1-JOB100-35 1509 ms
      

      Workaround

      None

            851f15845f55 Mateusz Szmal
            661435e617fd Vani
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated: