Low Issue Summary
REST API calls made to bambooBaseURL/rest/api/latest/agent/$agentID may lead to significant performance issues depending on the number of deployment environments and/or build jobs that the agents can execute. The memory pressure induced by such requests has the potential to render the application unresponsive, and in severe cases, trigger "java.lang.OutOfMemoryError: Java heap space" errors.
This is reproducible on Data Center: 
Steps to Reproduce
- Create a large number of deployment environments that can be deployed by a single agent (for example, 20.000).
- This can be easily accomplished using Java Specs, especially for testing purposes.
- Turn on GC logging or watch memory usage with a Java profiler (for example, JProfiler, Java VisualVM, etc).
- Make a handful of GET requests against bambooBaseURL/rest/api/latest/agent/$agentID by visiting this url using multiple tabs on a web browser or triggering cURL requests.
Expected Results
Bamboo doesn't try to load all the data into memory in one go. So, when you make a request, it loads up smoothly, whether you're using a web browser or going the cURL route, based on the examples mentioned above.
Actual Results
Watch as memory pressure starts to rise in GC logging or your Java profiler. Depending on the number of requests you're firing off, the amount of deployment environments the agent can run, and the heap size assigned to the Bamboo JVM, you might notice things slowing down a bit, going unresponsive temporarily, or even crashing with "java.lang.OutOfMemoryError: Java heap space" errors.
A heap dump would reveal a significant portion of the heap memory being occupied by database query results generated from the threads responsible for processing the bambooBaseURL/rest/api/latest/agent/$agentID requests.
The threads handling the bambooBaseURL/rest/api/latest/agent/$agentID requests in thread dumps may run for extended periods, ranging from several minutes to hours, exhibiting the stack traces below:
"http-nio-8085-exec-10691" #1819149 daemon prio=5 os_prio=0 cpu=357889.52ms elapsed=6746.66s tid=0x00007fab8c0a9000 nid=0x10a8d4 runnable [0x00007fa67bff5000]
java.lang.Thread.State: RUNNABLE
at org.hibernate.engine.internal.EntityEntryContext.downgradeLocks(EntityEntryContext.java:375)
at org.hibernate.engine.internal.StatefulPersistenceContext.afterTransactionCompletion(StatefulPersistenceContext.java:309)
at org.hibernate.internal.SessionImpl.afterTransactionCompletion(SessionImpl.java:2461)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.afterTransactionCompletion(JdbcCoordinatorImpl.java:456)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.afterCompletionCallback(JdbcResourceLocalTransactionCoordinatorImpl.java:203)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.access$400(JdbcResourceLocalTransactionCoordinatorImpl.java:40)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:283)
at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:101)
at org.springframework.orm.hibernate5.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:645)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:743)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:711)
at jdk.internal.reflect.GeneratedMethodAccessor158.invoke(Unknown Source)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.23/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@11.0.23/Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:234)
at com.sun.proxy.$Proxy147.commit(Unknown Source)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:654)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:407)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241)
at com.sun.proxy.$Proxy295.findById(Unknown Source)
at com.atlassian.bamboo.deployments.configuration.service.EnvironmentCustomConfigServiceImpl.getConfigEntity(EnvironmentCustomConfigServiceImpl.java:118)
at com.atlassian.bamboo.deployments.configuration.service.EnvironmentCustomConfigServiceImpl.getDockerPipelineConfiguration(EnvironmentCustomConfigServiceImpl.java:76)
at com.atlassian.bamboo.plan.ExecutableAgentsHelperImpl.getExecutableEnvironments(ExecutableAgentsHelperImpl.java:567)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.23/Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.23/NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.23/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@11.0.23/Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:234)
at com.sun.proxy.$Proxy329.getExecutableEnvironments(Unknown Source)
at com.atlassian.bamboo.agent.AgentExecutableServiceImpl.fetchExecutableEnvironments(AgentExecutableServiceImpl.java:62)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.23/Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.23/NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.23/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@11.0.23/Method.java:566)
at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
at com.sun.proxy.$Proxy524.fetchExecutableEnvironments(Unknown Source)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.23/Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.23/NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.23/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@11.0.23/Method.java:566)
...
"http-nio-8085-exec-10582" #1757398 daemon prio=5 os_prio=0 cpu=3060297.88ms elapsed=8848.71s tid=0x00007fab8c149000 nid=0xdf29c runnable [0x00007fa674cf5000]
java.lang.Thread.State: RUNNABLE
at org.hibernate.engine.internal.EntityEntryContext.downgradeLocks(EntityEntryContext.java:375)
at org.hibernate.engine.internal.StatefulPersistenceContext.afterTransactionCompletion(StatefulPersistenceContext.java:309)
at org.hibernate.internal.SessionImpl.afterTransactionCompletion(SessionImpl.java:2461)
at org.hibernate.engine.jdbc.internal.JdbcCoordinatorImpl.afterTransactionCompletion(JdbcCoordinatorImpl.java:456)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.afterCompletionCallback(JdbcResourceLocalTransactionCoordinatorImpl.java:203)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl.access$400(JdbcResourceLocalTransactionCoordinatorImpl.java:40)
at org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.commit(JdbcResourceLocalTransactionCoordinatorImpl.java:283)
at org.hibernate.engine.transaction.internal.TransactionImpl.commit(TransactionImpl.java:101)
at org.springframework.orm.hibernate5.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:645)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:743)
at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:711)
at jdk.internal.reflect.GeneratedMethodAccessor158.invoke(Unknown Source)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.23/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@11.0.23/Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:234)
at com.sun.proxy.$Proxy147.commit(Unknown Source)
at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:654)
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:407)
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:241)
at com.sun.proxy.$Proxy232.getPlanById(Unknown Source)
at com.atlassian.bamboo.buildqueue.manager.AgentAssignmentMapImpl.checkAssignmentRequirements(AgentAssignmentMapImpl.java:145)
at com.atlassian.bamboo.buildqueue.manager.AgentAssignmentMapImpl.checkAssignmentRequirements(AgentAssignmentMapImpl.java:101)
at com.atlassian.bamboo.plan.ExecutableAgentsHelperImpl.getExecutableBuildables(ExecutableAgentsHelperImpl.java:627)
at com.atlassian.bamboo.plan.ExecutableAgentsHelperImpl.getExecutableBuildables(ExecutableAgentsHelperImpl.java:545)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.23/Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.23/NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.23/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@11.0.23/Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:234)
at com.sun.proxy.$Proxy329.getExecutableBuildables(Unknown Source)
at com.atlassian.bamboo.agent.AgentExecutableServiceImpl.fetchExecutableBuildables(AgentExecutableServiceImpl.java:55)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.23/Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.23/NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.23/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@11.0.23/Method.java:566)
at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
at com.sun.proxy.$Proxy524.fetchExecutableBuildables(Unknown Source)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(java.base@11.0.23/Native Method)
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(java.base@11.0.23/NativeMethodAccessorImpl.java:62)
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.23/DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(java.base@11.0.23/Method.java:566)
...
Workaround
Currently there is no known workaround for this behaviour. A workaround will be added here when available.