Uploaded image for project: 'Bamboo Data Center'
  1. Bamboo Data Center
  2. BAM-25635

ActiveMQ will not shutdown completely when the primary node hangs due to a Lockup/Hardware/Hypervisor failure

XMLWordPrintable

      Issue Summary

      This is reproducible on Data Center:

      An active Bamboo node will not shutdown ActiveMQ correctly in case of a Hardware/Hypervisor lockup and will hold the Java PID in a running state.

      Steps to Reproduce

      1. Run Bamboo
      2. Simulate a hardware/VM lockup (suspend/unsuspend) or use Docker pause/unpause
      3. Observe in the logs that Bamboo will commence a shutdown
      4. Notice that Bamboo can't shutdown completely reporting a new ActiveMQ thread

      Expected Results

      Bamboo should finish the shutdown process completely.

      Actual Results

      A message similar to the following is prompted in the logs. The Bamboo Java PID will not exit.

      2024-01-10 10:24:42,743 INFO [ActiveMQ ShutdownHook] [BrokerPluginSupport] Broker Plugin org.apache.activemq.broker.util.TimeStampingBrokerPlugin stopped
2024-01-10 10:24:42,743 INFO [ActiveMQ ShutdownHook] [PListStoreImpl] PListStore:[/bamboo-shared-home/jms-store/bamboo/tmp_storage] stopped
2024-01-10 10:24:42,744 INFO [ActiveMQ ShutdownHook] [KahaDBStore] Stopping async queue tasks
2024-01-10 10:24:42,744 INFO [ActiveMQ ShutdownHook] [KahaDBStore] Stopping async topic tasks
2024-01-10 10:24:42,744 INFO [ActiveMQ ShutdownHook] [KahaDBStore] Stopped KahaDB
2024-01-10 10:24:42,764 INFO [ActiveMQ ShutdownHook] [BambooAmqClusterLocker] Bamboo amq cluster locker stopped
2024-01-10 10:24:42,820 INFO [ActiveMQ ShutdownHook] [BrokerService] Apache ActiveMQ 5.18.3 (bamboo, ID:d-bamboo950-44931-1704783487689-0:1) uptime 16 hours 26 minutes
2024-01-10 10:24:42,820 INFO [ActiveMQ ShutdownHook] [BrokerService] Apache ActiveMQ 5.18.3 (bamboo, ID:d-bamboo950-44931-1704783487689-0:1) is shutdown
2024-01-10 10:24:42,825 INFO [ActiveMQ Connection Executor: vm://bamboo#0] [BrokerService] Using Persistence Adapter: KahaDBPersistenceAdapter[/activemq-data/bamboo/KahaDB]
2024-01-10 10:24:42,825 INFO [ActiveMQ Connection Executor: vm://bamboo#0] [BrokerService] Starting Persistence Adapter: KahaDBPersistenceAdapter[/activemq-data/bamboo/KahaDB]
2024-01-10 10:24:42,848 INFO [ActiveMQ Connection Executor: vm://bamboo#0] [SharedFileLocker] Database activemq-data/bamboo/KahaDB/lock is locked by another server. This broker is now in slave mode waiting a lock to be acquired

      Workaround

      Currently there is no known workaround for this behavior. A workaround will be added here when available

          Form Name

            [BAM-25635] ActiveMQ will not shutdown completely when the primary node hangs due to a Lockup/Hardware/Hypervisor failure

              73868399605e Eduardo Alvarenga (Inactive)
              73868399605e Eduardo Alvarenga (Inactive)
              Affected customers:
              0 This affects my team
              Watchers:
              2 Start watching this issue

                Created:
                Updated:
                Resolved: