Issue Summary

The remote-link-aggregation requests are causing performance issues due to the amount of data they have to gather/aggregate (e.g. find deployment results, releases, check permissions, group versions by project, and many others). It's a rather complex (CPU-intensive) process that demands a lot from the system and may lead to an application outage. The more deployment results, projects, environments, and etc you have in Bamboo the slower the aggregate function becomes.

In order to diagnose this issue look for the presence of long running HTTP threads that are calling the following REST endpoint while Bamboo's unavailable or reporting high CPU usage for extended periods inside the Bamboo access logs:

• /rest/remote-link-aggregation/latest/aggregation?globalId=

You can also look for the presence of such endpoints inside the catalina log files reported by the StuckThreadDetectionValve as (potentially) being stuck or having completed after several minutes or even hours:

11-Aug-2022 05:35:23.917 WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [http-nio-8085-exec-624] (id=[885617]) has been active for [75,034] milliseconds (since [8/11/22, 5:34 AM]) to serve the same request for [https://bamboo.com/rest/remote-link-aggregation/latest/aggregation?globalId=ABC-5996&globalId=ABC-5795&globalId=SUP-7752&globalId=RE-1&globalId=TEAM-1074&globalId=ABC-5945&globalId=ABC-5974&globalId=ABC-5943&globalId=ABC-5863&globalId=DEF-5589&globalId=ABC-5940&globalId=DEF-1071&globalId=DEF-1026&globalId=DEF-7018&globalId=ABC-5886&globalId=ABC-5813&globalId=ABC-6056&globalId=ABC-5697&globalId=ABC-5938&globalId=ABC-5872&globalId=ABC-5952&globalId=ABC-5688&globalId=ABC-5993&globalId=ABC-6031&globalId=ABC-6025&globalId=ABC-5875] and may be stuck (configured threshold for this StuckThreadDetectionValve is [60] seconds). There is/are [43] thread(s) in total that are monitored by this Valve and may be stuck.
        java.lang.Throwable
...
11-Aug-2022 07:01:40.976 WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadCompleted Thread [http-nio-8085-exec-624] (id=[885617]) was previously reported to be stuck but has completed. It was active for approximately [5,252,011] milliseconds. There is/are still [140] thread(s) that are monitored by this Valve and may be stuck.

Alternatively you can take a set of thread dumps while Bamboo's unavailable or reporting high CPU usage for extended periods and look for the presence of several HTTP threads spending a long processing data with the following class in their stack:

com.atlassian.bamboo.plugins.jira.summary.deployment.DeploymentAggregator.aggregateForGlobalId

Such threads containing the class above might be attempting to fulfil the following types of requests (incl. but not limited to):

• getRelatedPlanResultKeys
• getEnvironmentsForDeploymentProject
• getDeploymentVersion
• getLatestKnownDeploymentResultDtoForEnvironment

This is reproducible on Data Center: Yes.

Steps to Reproduce

N/A

Expected Results

Bamboo does not spend too much time trying to gather/aggregate the data requested by the DeploymentAggregator function.

Actual Results

Bamboo spends a lot of time and resources trying to fetch the data requested by the DeploymentAggregator function leading to a serious performance degradation or even a full outage.

Workaround

The REST call (remote-link-aggregation) causing problems is part of the integration between Bamboo and Jira which is handled by an app called Atlassian Bamboo Jira Plugin.

Aggregation is just one of the functionalities of the Atlassian Bamboo Jira Plugin app. This isn't a permanent solution but you can temporarily turn off this feature of the app by following these steps:

1. Access the Bamboo administration > Manage apps page.
2. Select All apps in the dropdown field close to the search box.
3. Search for Atlassian Bamboo Jira Plugin.
4. Click on the app (to load its details) and expand its 10 modules.
5. Disable the following modules:
   • deployment-remote-link-aggregator
   • Development Status Summary Capability (devStatusSummaryCapability)

The load on your Bamboo instance/server should go down after that and for as long as the modules above remain disabled.