Details
-
Bug
-
Resolution: Won't Fix
-
Low
-
None
-
3.1.4
-
None
-
CentOS 5.5, JDK 1.6.0_26
Description
Today we experienced an issue where our Bamboo server ran out of disk space.
It came to our attention due to to the fact that all of the remote agents went offline at the same time.
We had set up the agent wrapper.conf to continue to try and reconnect to the server for 6 hours after an initial connection failure.
When the server ran out of disk space we saw the following in the agent logs:
INFO | jvm 1 | 2011/08/25 03:25:05 | org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device INFO | jvm 1 | 2011/08/25 03:25:05 | Caused by: javax.jms.JMSException: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.ActiveMQConnection.doSyncSendPacket(ActiveMQConnection.java:1329) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1314) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1758) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:231) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:74) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.apache.activemq.pool.PooledProducer.send(PooledProducer.java:59) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:536) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:513) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate$2.doInJms(JmsTemplate.java:479) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:430) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:477) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:588) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:576) INFO | jvm 1 | 2011/08/25 03:25:05 | at com.atlassian.bamboo.v2.build.agent.remote.sender.JmsTemplateMessageSender.send(JmsTemplateMessageSender.java:33) INFO | jvm 1 | 2011/08/25 03:25:05 | at com.atlassian.bamboo.v2.build.agent.remote.heartbeat.AgentHeartBeatJob.execute(AgentHeartBeatJob.java:51) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.quartz.core.JobRunShell.run(JobRunShell.java:202) INFO | jvm 1 | 2011/08/25 03:25:05 | at com.atlassian.bamboo.quartz.SystemAuthorizedThreadPool$1.run(SystemAuthorizedThreadPool.java:38) INFO | jvm 1 | 2011/08/25 03:25:05 | at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525) INFO | jvm 1 | 2011/08/25 03:25:05 | Caused by: java.lang.Throwable: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device
Even after we freed 50GB on the server disk, we saw the following in the agent logs:
INFO | jvm 86 | 2011/08/25 10:31:31 | 2011-08-25 10:31:31,663 INFO [QuartzScheduler_Worker-2] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@2d20d239' was unsuccessful. Run 1 / 15. Exception thrown when running task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@2d20d239', with message: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device INFO | jvm 86 | 2011/08/25 10:31:31 | 2011-08-25 10:31:31,664 INFO [QuartzScheduler_Worker-2] [RetryingTaskExecutor] Waiting 1000ms before retrying... INFO | jvm 86 | 2011/08/25 10:31:32 | 2011-08-25 10:31:32,674 INFO [QuartzScheduler_Worker-2] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@2d20d239' was unsuccessful. Run 2 / 15. Exception thrown when running task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@2d20d239', with message: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: java.util.concurrent.ExecutionException: java.io.IOException: No space left on device INFO | jvm 86 | 2011/08/25 10:31:32 | 2011-08-25 10:31:32,674 INFO [QuartzScheduler_Worker-2] [RetryingTaskExecutor] Waiting 2000ms before retrying... INFO | jvm 86 | 2011/08/25 10:31:34 | 2011-08-25 10:31:34,683 INFO [QuartzScheduler_Worker-2] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@2d20d239' finished successfully during retry number 2 INFO | jvm 86 | 2011/08/25 11:07:16 | 2011-08-25 11:07:16,646 INFO [QuartzScheduler_Worker-10] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, dropping the current heartbeat... INFO | jvm 86 | 2011/08/25 11:10:46 | 2011-08-25 11:10:46,650 INFO [QuartzScheduler_Worker-7] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, dropping the current heartbeat... INFO | jvm 86 | 2011/08/25 11:10:51 | 2011-08-25 11:10:51,647 INFO [QuartzScheduler_Worker-1] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, dropping the current heartbeat... INFO | jvm 86 | 2011/08/25 11:12:01 | 2011-08-25 11:12:01,651 INFO [QuartzScheduler_Worker-6] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, dropping the current heartbeat...
Attempting to restart one of the agents we saw:
INFO | jvm 2 | 2011/08/25 04:29:09 | 2011-08-25 04:29:09,370 WARN [Thread-1] [JmsClientInterceptor] Remote access error: ReflectiveMethodInvocation: public abstract com.atlassian.bamboo.buildqueue.PipelineDefinition com.atlassian.bamboo.buildqueue.manager.RemoteAgentManager.registerAgent(com.atlassian.bamboo.buildqueue.PipelineDefinition) throws com.atlassian.bamboo.license.BambooLicenseException; target is null INFO | jvm 2 | 2011/08/25 04:29:09 | javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.createJMSException(MultiplexingRequestor.java:205) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.request(MultiplexingRequestor.java:133) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.lingo.BambooMultiplexingRequestor.request(BambooMultiplexingRequestor.java:66) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.JmsClientInterceptor.invoke(JmsClientInterceptor.java:138) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) INFO | jvm 2 | 2011/08/25 04:29:09 | at $Proxy10.registerAgent(Unknown Source) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.AgentRegistrationBean.registerAgent(AgentRegistrationBean.java:41) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) INFO | jvm 2 | 2011/08/25 04:29:09 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1237) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1203) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1167) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:427) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:249) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:155) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:246) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:160) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:291) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:352) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:122) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:88) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.initApplicationContext(RemoteAgent.java:170) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.start(RemoteAgent.java:52) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.agent.bootstrap.AgentRunner.run(AgentRunner.java:21) INFO | jvm 2 | 2011/08/25 04:29:09 | at java.lang.Thread.run(Thread.java:662) INFO | jvm 2 | 2011/08/25 04:29:09 | Exiting due to fatal exception. INFO | jvm 2 | 2011/08/25 04:29:09 | org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'agentRegistrationBean' defined in class path resource [applicationContextRemoteAgentStartup.xml]: Invocation of init method failed; nested exception is org.springframework.remoting.RemoteAccessException: Cannot access JMS invoker remote service at [null]; nested exception is javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | Caused by: org.springframework.remoting.RemoteAccessException: Cannot access JMS invoker remote service at [null]; nested exception is javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | Caused by: javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.createJMSException(MultiplexingRequestor.java:205) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.request(MultiplexingRequestor.java:133) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.lingo.BambooMultiplexingRequestor.request(BambooMultiplexingRequestor.java:66) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.JmsClientInterceptor.invoke(JmsClientInterceptor.java:138) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) INFO | jvm 2 | 2011/08/25 04:29:09 | at $Proxy10.registerAgent(Unknown Source) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.AgentRegistrationBean.registerAgent(AgentRegistrationBean.java:41) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) INFO | jvm 2 | 2011/08/25 04:29:09 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1237) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1203) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1167) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:427) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:249) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:155) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:246) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:160) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:291) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:352) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:122) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:88) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.initApplicationContext(RemoteAgent.java:170) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.start(RemoteAgent.java:52) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.agent.bootstrap.AgentRunner.run(AgentRunner.java:21) INFO | jvm 2 | 2011/08/25 04:29:09 | at java.lang.Thread.run(Thread.java:662) INFO | jvm 2 | 2011/08/25 04:29:09 | 2011-08-25 04:29:09,371 FATAL [Thread-1] [AgentBootstrap] Exiting due to fatal exception. INFO | jvm 2 | 2011/08/25 04:29:09 | org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'agentRegistrationBean' defined in class path resource [applicationContextRemoteAgentStartup.xml]: Invocation of init method failed; nested exception is org.springframework.remoting.RemoteAccessException: Cannot access JMS invoker remote service at [null]; nested exception is javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | Caused by: org.springframework.remoting.RemoteAccessException: Cannot access JMS invoker remote service at [null]; nested exception is javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | Caused by: javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | Caused by: javax.jms.JMSException: edu.emory.mathcs.backport.java.util.concurrent.TimeoutException INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.createJMSException(MultiplexingRequestor.java:205) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.impl.MultiplexingRequestor.request(MultiplexingRequestor.java:133) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.lingo.BambooMultiplexingRequestor.request(BambooMultiplexingRequestor.java:66) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.logicblaze.lingo.jms.JmsClientInterceptor.invoke(JmsClientInterceptor.java:138) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) INFO | jvm 2 | 2011/08/25 04:29:09 | at $Proxy10.registerAgent(Unknown Source) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.AgentRegistrationBean.registerAgent(AgentRegistrationBean.java:41) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) INFO | jvm 2 | 2011/08/25 04:29:09 | at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) INFO | jvm 2 | 2011/08/25 04:29:09 | at java.lang.reflect.Method.invoke(Method.java:597) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1237) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1203) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1167) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:427) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:249) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:155) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:246) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:160) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:291) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:352) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:122) INFO | jvm 2 | 2011/08/25 04:29:09 | at org.springframework.context.support.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:88) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.initApplicationContext(RemoteAgent.java:170) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.v2.build.agent.remote.RemoteAgent.start(RemoteAgent.java:52) INFO | jvm 2 | 2011/08/25 04:29:09 | at com.atlassian.bamboo.agent.bootstrap.AgentRunner.run(AgentRunner.java:21) INFO | jvm 2 | 2011/08/25 04:29:09 | at java.lang.Thread.run(Thread.java:662) ERROR | wrapper | 2011/08/25 04:29:09 | JVM exited unexpectedly.
It was only when we restarted the server that the agents reconnected successfully (we didn't restart any of the agents).
It seems that when the disk space error occurred and the agents all took themselves offline, Bamboo made it impossible for them to come back online even though they were trying continuously to do so. This seems like a bug relating to Bamboo's management of connection state for the remote agents.