-
Type:
Bug
-
Resolution: Fixed
-
Priority:
High
-
Affects Version/s: 9.2.7, 9.3.5, 9.4.1
-
Component/s: Build Queues
-
None
-
1
-
Severity 2 - Major
Problem
When a "No space left on device" occurs in Bamboo, ActiveMQ processes its shutdown successfully but the rest of the application remains up, causing Agents and builds to fail, though the UI is still accessible.
Environment
- 9.2.7, 9.3.5, 9.4.1
Steps to Reproduce
- Start Bamboo
- Fill the disk enough to create a "No space left on device" on the Bamboo server
- Wait for ActiveMQ to shutdown
- Observe the application remains up
Expected Results
Bamboo should follow the ActiveMQ shutdown and stop the application completely.
Actual Results
Only ActiveMQ is shutdown, leaving the rest of the application up and creating several errors due to the Queue, Agents and other operations not being operational as ActiveMQ is down.
2023-12-05 21:34:35,272 WARN [ActiveMQ Data File Writer] [DataFileAppender] Journal failed while writing at: 4723:7322177 java.io.IOException: No space left on device at java.io.RandomAccessFile.writeBytes(Native Method) ~[?:?] at java.io.RandomAccessFile.write(RandomAccessFile.java:559) ~[?:?] at org.apache.activemq.util.RecoverableRandomAccessFile.write(RecoverableRandomAccessFile.java:249) ~[activemq-kahadb-store-5.16.7.jar:5.16.7] at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender.processQueue(DataFileAppender.java:335) ~[activemq-kahadb-store-5.16.7.jar:5.16.7] at org.apache.activemq.store.kahadb.disk.journal.DataFileAppender$1.run(DataFileAppender.java:172) ~[activemq-kahadb-store-5.16.7.jar:5.16.7] 2023-12-05 21:34:35,269 INFO [8-BuildTailMessageProcessingTh2023-12-05 21:34:35,455 WARN [ActiveMQ NIO Worker 898559] [TransportConnection] Transport Connection to: tcp://10.99.99.10:58984 failed: Broker BrokerService[bamboo] is being stopped 2023-12-05 21:34:35,455 WARN [ActiveMQ NIO Worker 898596] [TransportConnection] Transport Connection to: tcp://10.99.99.10:58979 failed: Broker BrokerService[bamboo] is being stopped 2023-12-05 21:34:35,458 INFO [ActiveMQ Transport Server Thread Handler: nio://0.0.0.0:54663?wireFormat.maxInactivityDuration=300000] [TcpTransportServer$1] socketQueue interrupted - stopping 2023-12-05 21:34:35,459 INFO [ActiveMQ Transport Server Thread Handler: nio://0.0.0.0:54663?wireFormat.maxInactivityDuration=300000] [TransportConnector$1] Could not accept connection during shutdown : null (null) 2023-12-05 21:34:35,566 INFO [IOExceptionHandler: stopping BrokerService[bamboo]] [TransportConnector] Connector nio://bambooserver:54663?wireFormat.maxInactivityDuration=300000 stopped 2023-12-05 21:34:35,566 INFO [IOExceptionHandler: stopping BrokerService[bamboo]] [TransportConnector] Connector tcp://localhost:54665?wireFormat.maxInactivityDuration=300000 stopped 2023-12-05 21:34:35,567 INFO [IOExceptionHandler: stopping BrokerService[bamboo]] [TransportConnector] Connector ssl://bambooserver:54664?wireFormat.maxInactivityDuration=300000 stopped 2023-12-05 21:34:35,626 INFO [IOExceptionHandler: stopping BrokerService[bamboo]] [TransportConnector] Connector vm://bamboo stopped 2023-12-05 21:34:35,626 WARN [bambooHeartBeatMessageListenerConnector-2] [DefaultMessageListenerContainer] Setup of JMS message listener invoker failed for destination 'queue://com.atlassian.bamboo.heartbeatQueue' - trying to recover. Cause: The Consumer is closed 2023-12-05 21:34:35,626 WARN [bambooAgentMessageListenerConnector-1] [DefaultMessageListenerContainer] Setup of JMS message listener invoker failed for destination 'queue://com.atlassian.bamboo.serverQueue' - trying to recover. Cause: The Session is closed 2023-12-05 21:34:35,626 WARN [buildTailMessageListenerConnector-1] [DefaultMessageListenerContainer] Setup of JMS message listener invoker failed for destination 'queue://com.atlassian.bamboo.buildTailQueue' - trying to recover. Cause: The Session is closed 2023-12-05 21:34:35,635 INFO [IOExceptionHandler: stopping BrokerService[bamboo]] [BrokerPluginSupport] Broker Plugin org.apache.activemq.broker.util.TimeStampingBrokerPlugin stopped 2023-12-05 21:34:35,636 INFO [IOExceptionHandler: stopping BrokerService[bamboo]] [PListStoreImpl] PListStore:[/data/bamboo-home/shared/jms-store/bamboo/tmp_storage] stopped 2023-12-05 21:34:35,636 INFO [IOExceptionHandler: stopping BrokerService[bamboo]] [KahaDBStore] Stopping async queue tasks 2023-12-05 21:34:35,636 INFO [IOExceptionHandler: stopping BrokerService[bamboo]] [KahaDBStore] Stopping async topic tasks 2023-12-05 21:34:35,636 INFO [IOExceptionHandler: stopping BrokerService[bamboo]] [KahaDBStore] Stopped KahaDB 2023-12-05 21:34:35,715 INFO [IOExceptionHandler: stopping BrokerService[bamboo]] [BambooAmqClusterLocker] Bamboo amq cluster locker stopped 2023-12-05 21:34:35,720 INFO [IOExceptionHandler: stopping BrokerService[bamboo]] [BrokerService] Apache ActiveMQ 5.16.7 (bamboo, ID:mnlbambwbp01-39139-1701064182159-0:1) uptime 8 days 6 hours 2023-12-05 21:34:35,720 INFO [IOExceptionHandler: stopping BrokerService[bamboo]] [BrokerService] Apache ActiveMQ 5.16.7 (bamboo, ID:mnlbambwbp01-39139-1701064182159-0:1) is shutdown 2023-12-05 21:34:35,848 INFO [ActiveMQ Connection Executor: vm://bamboo#0] [BrokerService] Using Persistence Adapter: KahaDBPersistenceAdapter[/activemq-data/bamboo/KahaDB] 2023-12-05 21:34:35,887 INFO [ActiveMQ Connection Executor: vm://bamboo#0] [SharedFileLocker] Database activemq-data/bamboo/KahaDB/lock is locked by another server. This broker is now in slave mode waiting a lock to be acquired
Workaround
Add a proper filesystem monitoring and avoid disk space events.
Notes
- mentioned in
-
Page Loading...