Uploaded image for project: 'Bamboo Data Center'
  1. Bamboo Data Center
  2. BAM-9941

ActiveMQ JMS data file no longer found causing builds not able to launch

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • Medium
    • 4.2
    • 3.3 final
    • Agents, Infrastructure

    Description

      BEAC server log JMS summary snipppet
      2011-10-09 18:34:29,222 ERROR [BrokerService[bamboo] Task] [Queue] Failed to page in more queue messages
      java.lang.RuntimeException: java.lang.RuntimeException: java.io.IOException: Could not locate data file /opt/j2ee/domains/bamboo.atlassian.com/confluence/webapps/atlassian-bamboo/data/current/jms-store/bamboo/KahaDB/db-1782.log
              at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:113)
              at org.apache.activemq.broker.region.cursors.StoreQueueCursor.reset(StoreQueueCursor.java:157)
              at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1677)
              at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1897)
              at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1424)
              at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:122)
              at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:43)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
              at java.lang.Thread.run(Thread.java:662)
      Caused by: java.lang.RuntimeException: java.io.IOException: Could not locate data file /opt/j2ee/domains/bamboo.atlassian.com/confluence/webapps/atlassian-bamboo/data/current/jms-store/bamboo/KahaDB/db-1782.log
              at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:265)
              at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:110)
              ... 9 more
      Caused by: java.io.IOException: Could not locate data file /opt/j2ee/domains/bamboo.atlassian.com/confluence/webapps/atlassian-bamboo/data/current/jms-store/bamboo/KahaDB/db-1782.log
              at org.apache.kahadb.journal.Journal.getDataFile(Journal.java:345)
              at org.apache.kahadb.journal.Journal.read(Journal.java:592)
              at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:786)
              at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:956)
              at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:494)
              at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
              at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recoverNextMessages(KahaDBStore.java:485)
              at org.apache.activemq.store.ProxyMessageStore.recoverNextMessages(ProxyMessageStore.java:88)
              at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.doFillBatch(QueueStorePrefetch.java:97)
              at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:262)
              ... 10 more
      2011-10-09 18:34:29,351 ERROR [BrokerService[bamboo] Task] [Journal] Looking for key 1782 but not found in fileMap: {1783=db-1783.log number = 1783 , length = 33554836, 1786=db-1786.log number = 1786 , length = 34109378, 1787=db-1787.log number = 1787 , length = 33556132, 1280=db-1280.log number = 1280 , length = 33556132, 1784=db-1784.log number = 1784 , length = 33554712, 1790=db-1790.log number = 1790 , length = 33606358, 1791=db-1791.log number = 1791 , length = 35664711, 1788=db-1788.log number = 1788 , length = 33581527, 1789=db-1789.log number = 1789 , length = 33557879, 1806=db-1806.log number = 1806 , length = 33592825, 1807=db-1807.log number = 1807 , length = 16003513, 1804=db-1804.log number = 1804 , length = 33635071, 1805=db-1805.log number = 1805 , length = 33554607, 1802=db-1802.log number = 1802 , length = 33580009, 1803=db-1803.log number = 1803 , length = 33554674, 1800=db-1800.log number = 1800 , length = 33567294, 1801=db-1801.log number = 1801 , length = 33819685, 1798=db-1798.log number = 1798 , length = 33971508, 1279=db-1279.log number = 1279 , length = 33554625, 1799=db-1799.log number = 1799 , length = 33630766, 1796=db-1796.log number = 1796 , length = 33651278, 1797=db-1797.log number = 1797 , length = 45623364, 1794=db-1794.log number = 1794 , length = 33666339, 1795=db-1795.log number = 1795 , length = 34512258, 1792=db-1792.log number = 1792 , length = 35143715, 1793=db-1793.log number = 1793 , length = 33776959}
      
      
      
      remote agents unable to send heartbeats
      2011-10-10 09:55:43,655 INFO [QuartzScheduler_Worker-9] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 13 seconds ago, dropping the current heartbeat...
      2011-10-10 10:05:43,655 INFO [QuartzScheduler_Worker-6] [AgentHeartBeatJob] Not sending a new heartbeat since an old one is still being sent, last successful transmission time was 7 seconds ago, dropping the current heartbeat..
      

      Likely caused by these ActiveMQ Issues:

      https://issues.apache.org/jira/browse/AMQ-3310
      or:
      https://issues.apache.org/jira/browse/AMQ-3470

      Reported internally in:
      https://extranet.atlassian.com/jira/browse/BUILDENG-649
      https://extranet.atlassian.com/jira/browse/BUILDENG-651

      The JMS spam caused the server logs to roll over so I was unable to capture the beginning of the problem.

      Confirmed server side that db-1782.log in the JMS repo was not present

      Attachments

        Issue Links

          Activity

            People

              pbruski Przemek Bruski
              bjohnson Bryce Johnson [Atlassian]
              Votes:
              2 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: