Details
-
Bug
-
Resolution: Fixed
-
Low
-
5.6.0
Description
On OnDemand we've seen several cases of Agents staying online even though their associated instance is no longer available. However their licenses are not freed up, preventing new instances from being started. If a new instance is started the agent will not load and the following message is given to the user on the Elastic Images page:
Registration of elastic agent at instance i-####### failed with a message: Cannot register additional elastic agent. You have reached the limit for the number of remote agents you can have. Please upgrade your license.
Bamboo will show the following in the logs:
@4000000053db1b9a14afe40c 2014-08-01 06:46:08,347 WARN [QuartzScheduler_Worker-2] [RemoteAgentManagerImpl] Detected that remote agent 'Elastic Agent on i-######' has been inactive since Wed Jul 30 19:35:12 CEST 2014 @4000000053db1b9a14b05d24 2014-08-01 06:46:08,347 WARN [QuartzScheduler_Worker-2] [RemoteAgentManagerImpl] Remote agent 'Elastic Agent on i-#######' was unresponsive and has gone offline. @4000000053db1b9a14b936c4 2014-08-01 06:46:08,347 ERROR [QuartzScheduler_Worker-2] [JobRunShell] Job HeartBeatChecker.HeartBeatChecker threw an unhandled Exception: @4000000053db1b9a14b93aac java.lang.NullPointerException @4000000053db1b9a14b93e94 at com.atlassian.bamboo.deployments.runtime.DeploymentsInProgressServiceImpl.getDeploymentExecutingOnAgent(DeploymentsInProgressServiceImpl.java:103) @4000000053db1b9a14b9427c at com.atlassian.bamboo.buildqueue.manager.AgentManagerImpl.abandonDeployment(AgentManagerImpl.java:542) @4000000053db1b9a14b95604 at com.atlassian.bamboo.buildqueue.manager.AgentManagerImpl.abandonBuild(AgentManagerImpl.java:570) @4000000053db1b9a14b959ec at com.atlassian.bamboo.buildqueue.manager.AgentManagerImpl.stopAgent(AgentManagerImpl.java:612) @4000000053db1b9a14b95dd4 at sun.reflect.GeneratedMethodAccessor4078.invoke(Unknown Source) @4000000053db1b9a14b961bc at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) @4000000053db1b9a14b9698c at java.lang.reflect.Method.invoke(Method.java:597) @4000000053db1b9a14b96d74 at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:304) @4000000053db1b9a14b9715c at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182) @4000000053db1b9a14b980fc at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149) @4000000053db1b9a14b988cc at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106) @4000000053db1b9a14b98cb4 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) @4000000053db1b9a14b99484 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) @4000000053db1b9a14b9986c at $Proxy87.stopAgent(Unknown Source) @4000000053db1b9a14b99c54 at com.atlassian.bamboo.buildqueue.manager.RemoteAgentManagerImpl.stopRemoteAgent(RemoteAgentManagerImpl.java:492) @4000000053db1b9a14b9a03c at com.atlassian.bamboo.buildqueue.manager.RemoteAgentManagerImpl.checkOfflineAgents(RemoteAgentManagerImpl.java:426) @4000000053db1b9a14b9c364 at com.atlassian.bamboo.v2.build.agent.HeartBeatCheckerJob.execute(HeartBeatCheckerJob.java:26) @4000000053db1b9a14b9c74c at org.quartz.core.JobRunShell.run(JobRunShell.java:202) @4000000053db1b9a14b9cb34 at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:49) @4000000053db1b9a14b9cf1c at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:31) @4000000053db1b9a14b9debc at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:20) @4000000053db1b9a14b9e2a4 at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:52) @4000000053db1b9a14b9e68c at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525) @4000000053db1b9a14bc960c 2014-08-01 06:46:08,347 ERROR [QuartzScheduler_Worker-2] [ErrorLogger] Job (HeartBeatChecker.HeartBeatChecker threw an exception. @4000000053db1b9a14bc99f4 org.quartz.SchedulerException: Job threw an unhandled exception. [See nested exception: java.lang.NullPointerException] @4000000053db1b9a14bc9ddc at org.quartz.core.JobRunShell.run(JobRunShell.java:213) @4000000053db1b9a14bca1c4 at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:49) @4000000053db1b9a14bca5ac at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:31) @4000000053db1b9a14bcb164 at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:20) @4000000053db1b9a14bcb54c at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:52) @4000000053db1b9a14bcb934 at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525) @4000000053db1b9a14bcb934 Caused by: java.lang.NullPointerException @4000000053db1b9a14bcc104 at com.atlassian.bamboo.deployments.runtime.DeploymentsInProgressServiceImpl.getDeploymentExecutingOnAgent(DeploymentsInProgressServiceImpl.java:103) @4000000053db1b9a14bcc8d4 at com.atlassian.bamboo.buildqueue.manager.AgentManagerImpl.abandonDeployment(AgentManagerImpl.java:542) @4000000053db1b9a14bcd48c at com.atlassian.bamboo.buildqueue.manager.AgentManagerImpl.abandonBuild(AgentManagerImpl.java:570) @4000000053db1b9a14bcd874 at com.atlassian.bamboo.buildqueue.manager.AgentManagerImpl.stopAgent(AgentManagerImpl.java:612) @4000000053db1b9a14bcdc5c at sun.reflect.GeneratedMethodAccessor4078.invoke(Unknown Source) @4000000053db1b9a14bce044 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) @4000000053db1b9a14bd0f24 at java.lang.reflect.Method.invoke(Method.java:597) @4000000053db1b9a14bd130c at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:304) @4000000053db1b9a14bd16f4 at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182) @4000000053db1b9a14bd1adc at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149) @4000000053db1b9a14bd22ac at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106) @4000000053db1b9a14bd2694 at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) @4000000053db1b9a14bd2e64 at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) @4000000053db1b9a14bd3e04 at $Proxy87.stopAgent(Unknown Source) @4000000053db1b9a14bd41ec at com.atlassian.bamboo.buildqueue.manager.RemoteAgentManagerImpl.stopRemoteAgent(RemoteAgentManagerImpl.java:492) @4000000053db1b9a14bd45d4 at com.atlassian.bamboo.buildqueue.manager.RemoteAgentManagerImpl.checkOfflineAgents(RemoteAgentManagerImpl.java:426) @4000000053db1b9a14bd49bc at com.atlassian.bamboo.v2.build.agent.HeartBeatCheckerJob.execute(HeartBeatCheckerJob.java:26) @4000000053db1b9a14bd518c at org.quartz.core.JobRunShell.run(JobRunShell.java:202) @4000000053db1b9a14bd5574 ... 5 more
Workaround
Log a ticket with support and ask them to restart Bamboo to release the agents.