Details
-
Bug
-
Resolution: Support Request
-
Low
-
None
-
5.8.1
-
None
-
None
Description
We got some unexpected shutdowns of EC2 remote agents while a build job is running. The Bamboo server is located in a datacenter in Germany and the remote agents are running in EC2 region Ireland (EU_WEST_1). The system error pages has two different kind of messages for this event.
First: Agent 'Elastic Agent on i-9b925636' went offline while building OC-JS-JOB1-1386. The results of that build will not be available.
Second:
com.atlassian.bamboo.task.TaskException: Failed to execute task at com.atlassian.bamboo.plugins.ant.task.AntBuildTask.execute(AntBuildTask.java:106) at com.atlassian.bamboo.task.TaskExecutorImpl$3.call(TaskExecutorImpl.java:281) at com.atlassian.bamboo.task.TaskExecutorImpl$3.call(TaskExecutorImpl.java:278) at com.atlassian.bamboo.task.TaskExecutorImpl.executeTaskWithPrePostActions(TaskExecutorImpl.java:198) at com.atlassian.bamboo.task.TaskExecutorImpl.executeTasks(TaskExecutorImpl.java:278) at com.atlassian.bamboo.task.TaskExecutorImpl.execute(TaskExecutorImpl.java:105) at com.atlassian.bamboo.build.pipeline.tasks.ExecuteBuildTask.call(ExecuteBuildTask.java:75) at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent.build(DefaultBuildAgent.java:188) at com.atlassian.bamboo.v2.build.agent.BuildAgentControllerImpl.waitAndPerformBuild(BuildAgentControllerImpl.java:112) at com.atlassian.bamboo.v2.build.agent.DefaultBuildAgent$1.run(DefaultBuildAgent.java:110) at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:49) at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:31) at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:20) at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:52) at java.lang.Thread.run(Thread.java:662) Caused by: org.springframework.remoting.RemoteAccessException: Could not access JMS invoker queue [queue://com.atlassian.bamboo.administrationConfigurationAccessorQueue]; nested exception is javax.jms.JMSException: org.apache.activemq.transport.RequestTimedOutIOException at org.springframework.jms.remoting.JmsInvokerClientInterceptor.convertJmsInvokerAccessException(JmsInvokerClientInterceptor.java:433) at org.springframework.jms.remoting.JmsInvokerClientInterceptor.invoke(JmsInvokerClientInterceptor.java:206) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207) at com.sun.proxy.$Proxy11.getAdministrationConfiguration(Unknown Source) at com.atlassian.bamboo.utils.BambooUrl.withBaseUrlFromConfiguration(BambooUrl.java:78) at com.atlassian.bamboo.variable.CustomVariableContextImpl.addVariables(CustomVariableContextImpl.java:526) at com.atlassian.bamboo.variable.CustomVariableContextImpl.getCommonBuildVariables(CustomVariableContextImpl.java:481) at com.atlassian.bamboo.variable.CustomVariableContextImpl.filterVariableContextByName(CustomVariableContextImpl.java:349) at com.atlassian.bamboo.variable.CustomVariableContextImpl.getVariables(CustomVariableContextImpl.java:262) at com.atlassian.bamboo.process.EnvironmentVariableAccessorImpl.getEnvironment(EnvironmentVariableAccessorImpl.java:85) at com.atlassian.bamboo.process.ProcessServiceImpl.createExternalProcess(ProcessServiceImpl.java:167) at com.atlassian.bamboo.process.ProcessServiceImpl.executeExternalProcess(ProcessServiceImpl.java:209) at com.atlassian.bamboo.plugins.ant.task.AntBuildTask.execute(AntBuildTask.java:76) ... 14 more Caused by: javax.jms.JMSException: org.apache.activemq.transport.RequestTimedOutIOException at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72) at org.apache.activemq.ActiveMQConnection.doSyncSendPacket(ActiveMQConnection.java:1459) at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1439) at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1809) at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:289) at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:224) at org.apache.activemq.jms.pool.PooledProducer.send(PooledProducer.java:77) at org.apache.activemq.jms.pool.PooledProducer.send(PooledProducer.java:60) at org.springframework.jms.remoting.JmsInvokerClientInterceptor.doExecuteRequest(JmsInvokerClientInterceptor.java:354) at org.springframework.jms.remoting.JmsInvokerClientInterceptor.executeRequest(JmsInvokerClientInterceptor.java:253) at org.springframework.jms.remoting.JmsInvokerClientInterceptor.invoke(JmsInvokerClientInterceptor.java:203) ... 26 more Caused by: org.apache.activemq.transport.RequestTimedOutIOException at org.apache.activemq.transport.FutureResponse.getResult(FutureResponse.java:54) at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:92) at org.apache.activemq.ActiveMQConnection.doSyncSendPacket(ActiveMQConnection.java:1446) ... 35 more
Attached screenshots from the error console and support zip.
Our guess is that the job took to long and the agent doesn't recognise the running job. This does not occur every time. If it happens, then a couple of agents going down. Like 2 up to n. Where can we find settings to change timeouts or connections related settings to improve the behaviour?
