Medium More cases of agents failing to shutdown after the server went offline
@400000004b840b0a16a2b35c 2010-02-23 11:06:08,379 INFO [Thread-1] [AgentHeartBeatJobScheduler] Scheduled AgentHeartBeatJobScheduler to run every 5s. Next run at Tue Feb 23 11:06:08 CST 2010 @400000004b840b0a1717b864 2010-02-23 11:06:08,386 INFO [Thread-1] [RemoteAgent] ********************************************************************************************************************************************** @400000004b840b0a1717c804 2010-02-23 11:06:08,387 INFO [Thread-1] [RemoteAgent] * * @400000004b840b0a1717cfd4 2010-02-23 11:06:08,387 INFO [Thread-1] [RemoteAgent] * Bamboo agent 'atlassian27.managed.contegix.com (2)' ready to receive builds. @400000004b840b0a17181624 2010-02-23 11:06:08,387 INFO [Thread-1] [RemoteAgent] * Remote Agent Home: /opt/bamboo-agents/bamboo-agent-2/bamboo-agent-home @400000004b840b0a17181a0c 2010-02-23 11:06:08,387 INFO [Thread-1] [RemoteAgent] * Broker URL: failover:(tcp://172.16.3.18:54669?wireFormat.maxInactivityDuration=300000)?initialReconnectDelay=15000&maxReconnectAttempts=10 @400000004b840b0a17182d94 2010-02-23 11:06:08,387 INFO [Thread-1] [RemoteAgent] * * @400000004b840b0a171850bc 2010-02-23 11:06:08,387 INFO [Thread-1] [RemoteAgent] ********************************************************************************************************************************************** @400000004b841bb932690974 2010-02-23 12:17:19,845 WARN [InactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@30cefd] [FailoverTransport] Transport failed to tcp://172.16.3.18:54669?wireFormat.maxInactivityDuration=300000 , attempting to automatically reconnect due to: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too long: /172.16.3.18:54669 @400000004b841e1f154e4ad4 2010-02-23 12:27:33,357 WARN [InactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@e5376a] [FailoverTransport] Transport failed to tcp://172.16.3.18:54669?wireFormat.maxInactivityDuration=300000 , attempting to automatically reconnect due to: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too long: /172.16.3.18:54669 @400000004b841f821bf45d4c 2010-02-23 12:33:28,467 ERROR [ActiveMQ Task] [FailoverTransport] Failed to connect to transport after: 10 attempt(s) @400000004b8423732697dc24 2010-02-23 12:50:17,646 WARN [QuartzScheduler_Worker-6] [JmsTemplateMessageSender] JmsException when trying to send message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@132c515. Retrying... @400000004b8423732697ebc4 org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Wire format negotiation timeout: peer did not send his wire format. @400000004b8423732697f394 Caused by: javax.jms.JMSException: Wire format negotiation timeout: peer did not send his wire format. @400000004b8423732698e9ac at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49) @400000004b8423732698ed94 at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1244) @400000004b8423732698f17c at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1659) @400000004b8423732698f564 at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:227) @400000004b842373269908ec at org.jencks.amqpool.PooledProducer.send(PooledProducer.java:73) @400000004b84237326990cd4 at org.jencks.amqpool.PooledProducer.send(PooledProducer.java:58) @400000004b842373269914a4 at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:536) @400000004b842373269933e4 at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:513) @400000004b842373269937cc at org.springframework.jms.core.JmsTemplate$2.doInJms(JmsTemplate.java:479) @400000004b84237326993bb4 at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:430) @400000004b84237326993f9c at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:477) @400000004b84237326994384 at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:588) @400000004b84237326995324 at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:576) @400000004b84237326995af4 at com.atlassian.bamboo.v2.build.agent.remote.sender.JmsTemplateMessageSender.send(JmsTemplateMessageSender.java:33) @400000004b84237326995edc at com.atlassian.bamboo.v2.build.agent.remote.heartbeat.AgentHeartBeatJob.execute(AgentHeartBeatJob.java:51) @400000004b842373269966ac at org.quartz.core.JobRunShell.run(JobRunShell.java:202) @400000004b84237326997264 at com.atlassian.bamboo.quartz.SystemAuthorizedThreadPool$1.run(SystemAuthorizedThreadPool.java:36) @400000004b8423732699764c at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525) @400000004b84237326997e1c Caused by: java.io.IOException: Wire format negotiation timeout: peer did not send his wire format. @400000004b84237326999d5c at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:94) @400000004b8423732699a144 at org.apache.activemq.transport.failover.FailoverTransport.restoreTransport(FailoverTransport.java:598) @400000004b8423732699a52c at org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:687) @400000004b8423732699acfc at org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:115) @400000004b8423732699c084 at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) @400000004b8423732699c46c at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) @400000004b8423732699c854 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) @400000004b8423732699d024 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) @400000004b8423732699e3ac at java.lang.Thread.run(Thread.java:619) @400000004b8423e5189a60c4 2010-02-23 12:52:11,411 ERROR [ActiveMQ Task] [FailoverTransport] Failed to connect to transport after: 10 attempt(s) @400000004b8423e5189a7064 2010-02-23 12:52:11,412 INFO [QuartzScheduler_Worker-6] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@132c515' was unsuccessful. Run 1 / 15. Exception thrown when running task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@132c515', with message: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Wire format negotiation timeout: peer did not send his wire format.; nested exception is java.io.IOException: Wire format negotiation timeout: peer did not send his wire format. @400000004b8423e5189bc824 2010-02-23 12:52:11,412 INFO [QuartzScheduler_Worker-6] [RetryingTaskExecutor] Waiting 1000ms before retrying... @400000004b8423f520ea5e64 2010-02-23 12:52:27,551 ERROR [ActiveMQ Task] [FailoverTransport] Failed to connect to transport after: 10 attempt(s) @400000004b8423f520f3fb54 2010-02-23 12:52:27,552 INFO [QuartzScheduler_Worker-6] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@132c515' was unsuccessful. Run 2 / 15. Exception thrown when running task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@132c515', with message: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Connection refused; nested exception is java.net.ConnectException: Connection refused @400000004b8423f520f5acec 2010-02-23 12:52:27,552 INFO [QuartzScheduler_Worker-6] [RetryingTaskExecutor] Waiting 2000ms before retrying... @400000004b8423fc28fc4964 2010-02-23 12:52:34,687 ERROR [ActiveMQ Task] [FailoverTransport] Failed to connect to transport after: 10 attempt(s) @400000004b8423fc2905de84 2010-02-23 12:52:34,688 INFO [QuartzScheduler_Worker-6] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@132c515' was unsuccessful. Run 3 / 15. Exception thrown when running task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@132c515', with message: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Connection refused; nested exception is java.net.ConnectException: Connection refused @400000004b8423fc29072e74 2010-02-23 12:52:34,688 INFO [QuartzScheduler_Worker-6] [RetryingTaskExecutor] Waiting 4000ms before retrying... @400000004b84240530eee4c4 2010-02-23 12:52:43,820 ERROR [ActiveMQ Task] [FailoverTransport] Failed to connect to transport after: 10 attempt(s) @400000004b84240530f8b094 2010-02-23 12:52:43,821 INFO [QuartzScheduler_Worker-6] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@132c515' was unsuccessful. Run 4 / 15. Exception thrown when running task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@132c515', with message: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Connection refused; nested exception is java.net.ConnectException: Connection refused @400000004b84240530f9ecfc 2010-02-23 12:52:43,821 INFO [QuartzScheduler_Worker-6] [RetryingTaskExecutor] Waiting 8000ms before retrying... @400000004b842412392c7b74 2010-02-23 12:52:56,958 ERROR [ActiveMQ Task] [FailoverTransport] Failed to connect to transport after: 10 attempt(s) @400000004b84241239537fbc 2010-02-23 12:52:56,960 INFO [QuartzScheduler_Worker-6] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@132c515' was unsuccessful. Run 5 / 15. Exception thrown when running task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@132c515', with message: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Connection refused; nested exception is java.net.ConnectException: Connection refused @400000004b8424123953a2e4 2010-02-23 12:52:56,961 INFO [QuartzScheduler_Worker-6] [RetryingTaskExecutor] Waiting 16000ms before retrying... @400000004b84242806156d2c 2010-02-23 12:53:18,100 ERROR [ActiveMQ Task] [FailoverTransport] Failed to connect to transport after: 10 attempt(s) @400000004b842428062f8cac 2010-02-23 12:53:18,103 INFO [QuartzScheduler_Worker-6] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@132c515' was unsuccessful. Run 6 / 15. Exception thrown when running task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@132c515', with message: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Connection refused; nested exception is java.net.ConnectException: Connection refused @400000004b84242806301d34 2010-02-23 12:53:18,103 INFO [QuartzScheduler_Worker-6] [RetryingTaskExecutor] Waiting 32000ms before retrying... @400000004b8432620c09c78c 2010-02-23 13:54:00,201 WARN [InactivityMonitor Async Task: java.util.concurrent.ThreadPoolExecutor$Worker@7f30f8] [FailoverTransport] Transport failed to tcp://172.16.3.18:54669?wireFormat.maxInactivityDuration=300000 , attempting to automatically reconnect due to: org.apache.activemq.transport.InactivityIOException: Channel was inactive for too long: /172.16.3.18:54669 @400000004b8433c51247310c 2010-02-23 13:59:55,306 ERROR [ActiveMQ Task] [FailoverTransport] Failed to connect to transport after: 10 attempt(s) @400000004b8433c51259d67c 2010-02-23 13:59:55,306 WARN [QuartzScheduler_Worker-3] [JmsTemplateMessageSender] JmsException when trying to send message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@d8ced6. Retrying... @400000004b8433c51259de4c org.springframework.jms.UncategorizedJmsException: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Wire format negotiation timeout: peer did not send his wire format. @400000004b8433c51259ea04 Caused by: javax.jms.JMSException: Wire format negotiation timeout: peer did not send his wire format. @400000004b8433c5125a0174 at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49) @400000004b8433c5125a055c at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1244) @400000004b8433c5125a0944 at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1659) @400000004b8433c5125a1114 at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:227) @400000004b8433c5125a1ccc at org.jencks.amqpool.PooledProducer.send(PooledProducer.java:73) @400000004b8433c5125a249c at org.jencks.amqpool.PooledProducer.send(PooledProducer.java:58) @400000004b8433c5125a2884 at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:536) @400000004b8433c5125a43dc at org.springframework.jms.core.JmsTemplate.doSend(JmsTemplate.java:513) @400000004b8433c5125a47c4 at org.springframework.jms.core.JmsTemplate$2.doInJms(JmsTemplate.java:479) @400000004b8433c5125a4bac at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:430) @400000004b8433c5125a4f94 at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:477) @400000004b8433c5125a537c at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:588) @400000004b8433c5125a631c at org.springframework.jms.core.JmsTemplate.convertAndSend(JmsTemplate.java:576) @400000004b8433c5125a6aec at com.atlassian.bamboo.v2.build.agent.remote.sender.JmsTemplateMessageSender.send(JmsTemplateMessageSender.java:33) @400000004b8433c5125a6ed4 at com.atlassian.bamboo.v2.build.agent.remote.heartbeat.AgentHeartBeatJob.execute(AgentHeartBeatJob.java:51) @400000004b8433c5125a76a4 at org.quartz.core.JobRunShell.run(JobRunShell.java:202) @400000004b8433c5125a825c at com.atlassian.bamboo.quartz.SystemAuthorizedThreadPool$1.run(SystemAuthorizedThreadPool.java:36) @400000004b8433c5125a8a2c at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:525) @400000004b8433c5125a8e14 Caused by: java.io.IOException: Wire format negotiation timeout: peer did not send his wire format. @400000004b8433c5125aad54 at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:94) @400000004b8433c5125ab13c at org.apache.activemq.transport.failover.FailoverTransport.restoreTransport(FailoverTransport.java:598) @400000004b8433c5125ab524 at org.apache.activemq.transport.failover.FailoverTransport.doReconnect(FailoverTransport.java:687) @400000004b8433c5125abcf4 at org.apache.activemq.transport.failover.FailoverTransport$2.iterate(FailoverTransport.java:115) @400000004b8433c5125acc94 at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122) @400000004b8433c5125ad464 at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43) @400000004b8433c5125ad84c at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) @400000004b8433c5125adc34 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) @400000004b8433c5125b5934 at java.lang.Thread.run(Thread.java:619) @400000004b8433c5128376bc 2010-02-23 13:59:55,310 INFO [QuartzScheduler_Worker-3] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@d8ced6' was unsuccessful. Run 1 / 15. Exception thrown when running task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@d8ced6', with message: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Wire format negotiation timeout: peer did not send his wire format.; nested exception is java.io.IOException: Wire format negotiation timeout: peer did not send his wire format. @400000004b8433c51284805c 2010-02-23 13:59:55,310 INFO [QuartzScheduler_Worker-3] [RetryingTaskExecutor] Waiting 1000ms before retrying... @400000004b84342105a044fc 2010-02-23 14:01:27,093 ERROR [ActiveMQ Task] [FailoverTransport] Failed to connect to transport after: 10 attempt(s) @400000004b84342105a8e404 2010-02-23 14:01:27,094 INFO [QuartzScheduler_Worker-3] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@d8ced6' was unsuccessful. Run 2 / 15. Exception thrown when running task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@d8ced6', with message: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Connection refused; nested exception is java.net.ConnectException: Connection refused @400000004b84342105a9f18c 2010-02-23 14:01:27,094 INFO [QuartzScheduler_Worker-3] [RetryingTaskExecutor] Waiting 2000ms before retrying... @400000004b8434280d9ff7ec 2010-02-23 14:01:34,228 ERROR [ActiveMQ Task] [FailoverTransport] Failed to connect to transport after: 10 attempt(s) @400000004b8434280da8cda4 2010-02-23 14:01:34,229 INFO [QuartzScheduler_Worker-3] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@d8ced6' was unsuccessful. Run 3 / 15. Exception thrown when running task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@d8ced6', with message: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Connection refused; nested exception is java.net.ConnectException: Connection refused @400000004b8434280da9d744 2010-02-23 14:01:34,229 INFO [QuartzScheduler_Worker-3] [RetryingTaskExecutor] Waiting 4000ms before retrying... @400000004b84343115bc2004 2010-02-23 14:01:43,364 ERROR [ActiveMQ Task] [FailoverTransport] Failed to connect to transport after: 10 attempt(s) @400000004b84343115d58404 2010-02-23 14:01:43,366 INFO [QuartzScheduler_Worker-3] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@d8ced6' was unsuccessful. Run 4 / 15. Exception thrown when running task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@d8ced6', with message: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Connection refused; nested exception is java.net.ConnectException: Connection refused @400000004b84343115d83f3c 2010-02-23 14:01:43,366 INFO [QuartzScheduler_Worker-3] [RetryingTaskExecutor] Waiting 8000ms before retrying... @400000004b84343e1ddf2dbc 2010-02-23 14:01:56,500 ERROR [ActiveMQ Task] [FailoverTransport] Failed to connect to transport after: 10 attempt(s) @400000004b84343e1de7ff8c 2010-02-23 14:01:56,501 INFO [QuartzScheduler_Worker-3] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@d8ced6' was unsuccessful. Run 5 / 15. Exception thrown when running task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@d8ced6', with message: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Connection refused; nested exception is java.net.ConnectException: Connection refused @400000004b84343e1de910fc 2010-02-23 14:01:56,501 INFO [QuartzScheduler_Worker-3] [RetryingTaskExecutor] Waiting 16000ms before retrying... @400000004b84345326239684 2010-02-23 14:02:17,639 ERROR [ActiveMQ Task] [FailoverTransport] Failed to connect to transport after: 10 attempt(s) @400000004b843453262c9f04 2010-02-23 14:02:17,640 INFO [QuartzScheduler_Worker-3] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@d8ced6' was unsuccessful. Run 6 / 15. Exception thrown when running task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@d8ced6', with message: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Connection refused; nested exception is java.net.ConnectException: Connection refused @400000004b843453262da8a4 2010-02-23 14:02:17,640 INFO [QuartzScheduler_Worker-3] [RetryingTaskExecutor] Waiting 32000ms before retrying... @400000004b8434782e67054c 2010-02-23 14:02:54,777 ERROR [ActiveMQ Task] [FailoverTransport] Failed to connect to transport after: 10 attempt(s) @400000004b8434782e6fd71c 2010-02-23 14:02:54,779 INFO [QuartzScheduler_Worker-3] [RetryingTaskExecutor] Task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@d8ced6' was unsuccessful. Run 7 / 15. Exception thrown when running task 'Sending message com.atlassian.bamboo.v2.build.agent.messages.UpdateHeartbeatMessage@d8ced6', with message: Uncategorized exception occured during JMS processing; nested exception is javax.jms.JMSException: Connection refused; nested exception is java.net.ConnectException: Connection refused @400000004b8434782e70e88c 2010-02-23 14:02:54,779 INFO [QuartzScheduler_Worker-3] [RetryingTaskExecutor] Waiting 60000ms before retrying... @400000004b84646e00087604 2010-02-23 17:27:32,000 WARN [ActiveMQ Transport: tcp:///172.16.3.18:54669] [FailoverTransport] Transport failed to tcp://172.16.3.18:54669?wireFormat.maxInactivityDuration=300000 , attempting to automatically reconnect due to: java.io.EOFException
