Agent's executable jobs screen can cause severe performance problems when there are a lot of jobs

Issue Summary

This affects Bamboo Data Center:

Agent's executable jobs screen can cause severe performance problems when there are a lot of jobs:

• From the Default plan configuration > $JOB > Requirements > Can build on X agent(s) > $AGENT > Executable jobs tab:
  • $BAMBOO_BASE_URL/agent/viewAgentExecutablePlans.action

• From the Bamboo administration > Agents > $AGENT > Executable jobs tab:
  • $BAMBOO_BASE_URL/agent/admin/ajax/viewAgentExecutableJobs.action

Steps to Reproduce

1. Create a large number of jobs, e.g. 250.000 that can be deployed by a single agent.
2. Visit the Executable jobs tab following one of the paths mentioned above.
3. Take some thread dumps right after accessing that page. Locate the thread responsible for serving the HTTP request made on step 2 when opening the Executable jobs tab:
   Example

   "http-nio-8085-exec-45834" #20962524 daemon prio=5 os_prio=0 tid=0x00007eff604b4800 nid=0x101bd runnable [0x00007effd87a9000]
      java.lang.Thread.State: RUNNABLE
   	at java.util.Arrays.copyOf(Arrays.java:3181)
   	at java.util.ArrayList.grow(ArrayList.java:265)
   	at java.util.ArrayList.ensureExplicitCapacity(ArrayList.java:239)
   	at java.util.ArrayList.ensureCapacityInternal(ArrayList.java:231)
   	at java.util.ArrayList.add(ArrayList.java:462)
   	at org.apache.commons.lang3.builder.EqualsBuilder. (EqualsBuilder.java:231)
   	at com.atlassian.bamboo.plan.cache.AbstractImmutablePlan.equals(AbstractImmutablePlan.java:338)
   	at java.util.ArrayList.remove(ArrayList.java:532)
   	at org.acegisecurity.afterinvocation.CollectionFilterer.getFilteredObject(CollectionFilterer.java:76)
   	at org.acegisecurity.afterinvocation.AclEntryAfterInvocationCollectionFilteringProvider.decide(AclEntryAfterInvocationCollectionFilteringProvider.java:127)
   	at org.acegisecurity.afterinvocation.AfterInvocationProviderManager.decide(AfterInvocationProviderManager.java:74)
   	at org.acegisecurity.intercept.AbstractSecurityInterceptor.afterInvocation(AbstractSecurityInterceptor.java:179)
   	at org.acegisecurity.intercept.method.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:68)
   	at com.atlassian.bamboo.security.acegi.intercept.aopalliance.AuthorityOverrideMethodSecurityInterceptor.invoke(AuthorityOverrideMethodSecurityInterceptor.java:28)
   	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
   	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:123)
   	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:388)
   	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119)
   	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
   	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
   	at com.sun.proxy.$Proxy182.getPlans(Unknown Source)
   	at com.atlassian.bamboo.plan.ExecutableAgentsHelperImpl.getExecutableBuildables(ExecutableAgentsHelperImpl.java:453)
   	at com.atlassian.bamboo.plan.ExecutableAgentsHelperImpl.getExecutableBuildables(ExecutableAgentsHelperImpl.java:384)
   	at sun.reflect.GeneratedMethodAccessor5506.invoke(Unknown Source)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
   	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
   	at com.sun.proxy.$Proxy293.getExecutableBuildables(Unknown Source)
   	at com.atlassian.bamboo.agent.AgentExecutableServiceImpl.fetchExecutableBuildables(AgentExecutableServiceImpl.java:50)
   	at com.atlassian.bamboo.configuration.agent.ViewAgent.fetchExecutableBuildables(ViewAgent.java:202)
   	at com.atlassian.bamboo.configuration.agent.ViewAgent$$Lambda$4966/776428569.get(Unknown Source)
   	at io.atlassian.fugue.Suppliers$MemoizingSupplier.get(Suppliers.java:175)
   	- locked <0x00007f080c400838> (a io.atlassian.fugue.Suppliers$MemoizingSupplier)
   	at com.atlassian.bamboo.configuration.agent.ViewAgent.getExecutableBuildables(ViewAgent.java:197)
   	at sun.reflect.GeneratedMethodAccessor5505.invoke(Unknown Source)
   	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
   	at java.lang.reflect.Method.invoke(Method.java:498)
   	... [redacted -- the stack is too big to fit in the comment]
   	at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:639)
   	at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
   	at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:882)
   	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1647)
   	at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
   	- locked <0x00007f080c401be0> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper)
   	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
   	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
   	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
   	at java.lang.Thread.run(Thread.java:748)
      Locked ownable synchronizers:
   	- <0x00007f08acc01900> (a org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker)
   

Expected Results

Paginated results that don't try to load all of the jobs at once.

Actual Results

That thread will be stuck serving the same HTTP request for several hours! Similar threads will start to pile up adding more pressure to the JVM (memory) and the database until eventually Bamboo either runs out of JDBC connections or OOM.

Workaround

1. Block the URLs from user access at your load balancer or educate user base to avoid those menus.
2. Reduce amount of jobs to help mitigate the situation.
3. Change Bamboo's templates to prevent users from trying to load up those menus:
   Workaround 3 – Instructions

   1. Stop Bamboo.

   /admin/agent/viewAgentExecutableJobs.ftl

   2. Create a backup of the <bamboo-install>/atlassian-bamboo/admin/agent/viewAgentExecutableJobs.ftl file.
   3. Remove the following line from the viewAgentExecutableJobs.ftl file:

   viewAgentExecutableJobs.ftl

   [@agt.executablePlans executableBuildables=executableBuildables /]
   

   /agent/viewAgentExecutablePlans.ftl

   4. Create a backup of the <bamboo-install>/atlassian-bamboo/agent/viewAgentExecutablePlans.ftl file.
   5. Remove the following line from the viewAgentExecutablePlans.ftl file:

   viewAgentExecutablePlans.ftl

   [@agt.executablePlans executableBuildables=executableBuildables /]
   

   6. Start Bamboo.

   The tabs will still be there inside the agent summary page after restarting Bamboo but they will show up empty if users click on them -– Bamboo will no longer try to load the list of jobs that agents are capable of running.

   It's important to remember that changes made to templates won't survive an upgrade. Therefore you'd need to make the same changes again after upgrading Bamboo –- unless, of course, you are upgrading to a version of Bamboo where this bug has been fixed.