Uploaded image for project: 'Bamboo Data Center'
  1. Bamboo Data Center
  2. BAM-15100

Lots of duplicate ActiveMQ messages stay in the queue (?)

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Low Low
    • 5.7.0
    • None
    • None

      For unknown reasons, the Bamboo server started to receive a lot duplicate ActiveMQ messages :

      mmeinhold@jira-bamboo:/etc/service/bamboo-server/log/logs$ zgrep "ActiveMQ BrokerService" @400000005433d75c1c0295ec.s 
      2014-10-07 11:40:59,467 INFO [ActiveMQ BrokerService[bamboo] Task-34655] [TempQueue] temp-queue://ID:ip-172-30-147-214-57871-1412639319561-1:1:1 on dispose, purge of 1 pending messages: org.apache.activemq.broker.region.cursors.VMPendingMessageCursor@73d93bfa
      2014-10-07 11:57:24,238 WARN [ActiveMQ BrokerService[bamboo] Task-35318] [AbstractStoreCursor] org.apache.activemq.broker.region.cursors.QueueStorePrefetch@467d603a:com.atlassian.bamboo.buildTailQueue,batchResetNeeded=false,storeHasMessages=true,size=1,cacheEnabled=true,maxBatchSize:2,hasSpace:true - cursor got duplicate: ID:ip-172-30-146-28-53258-1412639445366-1:1:9:1:33161, 4
      2014-10-07 11:57:24,238 WARN [ActiveMQ BrokerService[bamboo] Task-35318] [Queue] duplicate message from store ID:ip-172-30-146-28-53258-1412639445366-1:1:9:1:33161, redirecting for dlq processing
      2014-10-07 11:57:24,262 WARN [ActiveMQ BrokerService[bamboo] Task-35318] [AbstractStoreCursor] org.apache.activemq.broker.region.cursors.QueueStorePrefetch@467d603a:com.atlassian.bamboo.buildTailQueue,batchResetNeeded=false,storeHasMessages=true,size=1,cacheEnabled=true,maxBatchSize:2,hasSpace:true - cursor got duplicate: ID:ip-172-30-144-159-47160-1412639319990-1:1:9:1:34668, 4
      2014-10-07 11:57:24,262 WARN [ActiveMQ BrokerService[bamboo] Task-35318] [Queue] duplicate message from store ID:ip-172-30-144-159-47160-1412639319990-1:1:9:1:34668, redirecting for dlq processing
      2014-10-07 11:57:24,268 WARN [ActiveMQ BrokerService[bamboo] Task-35320] [AbstractStoreCursor] org.apache.activemq.broker.region.cursors.QueueStorePrefetch@467d603a:com.atlassian.bamboo.buildTailQueue,batchResetNeeded=false,storeHasMessages=true,size=1,cacheEnabled=true,maxBatchSize:2,hasSpace:true - cursor got duplicate: ID:agent-07-02-45716-1412646530061-1:1:10:1:29947, 4
      2014-10-07 11:57:24,268 WARN [ActiveMQ BrokerService[bamboo] Task-35320] [Queue] duplicate message from store ID:agent-07-02-45716-1412646530061-1:1:10:1:29947, redirecting for dlq processing
      2014-10-07 11:57:24,311 WARN [ActiveMQ BrokerService[bamboo] Task-35318] [AbstractStoreCursor] org.apache.activemq.broker.region.cursors.QueueStorePrefetch@467d603a:com.atlassian.bamboo.buildTailQueue,batchResetNeeded=false,storeHasMessages=true,size=1,cacheEnabled=true,maxBatchSize:2,hasSpace:true - cursor got duplicate: ID:agent-07-02-45716-1412646530061-1:1:10:1:29948, 4
      2014-10-07 11:57:24,311 WARN [ActiveMQ BrokerService[bamboo] Task-35318] [Queue] duplicate message from store ID:agent-07-02-45716-1412646530061-1:1:10:1:29948, redirecting for dlq processing
      2014-10-07 11:57:24,319 WARN [ActiveMQ BrokerService[bamboo] Task-35320] [AbstractStoreCursor] org.apache.activemq.broker.region.cursors.QueueStorePrefetch@467d603a:com.atlassian.bamboo.buildTailQueue,batchResetNeeded=false,storeHasMessages=true,size=1,cacheEnabled=true,maxBatchSize:2,hasSpace:true - cursor got duplicate: ID:ip-172-30-144-218-58062-1412653895028-1:1:10:1:25659, 4
      ...
      

      This seems to result in a permanent increase in the ActiveMQ baseline memory usage:

      I'm seeing this on JBAC only.

        1. activemq-brokerservice-1d.log.gz
          268 kB
        2. Banners and Alerts.png
          Banners and Alerts.png
          220 kB
        3. jbac-activemq-dlq-memory-usage-1d.png
          jbac-activemq-dlq-memory-usage-1d.png
          18 kB
        4. jbac-activemq-memory-1d.png
          jbac-activemq-memory-1d.png
          91 kB

              pbruski Przemek Bruski
              mmeinhold Martin Meinhold
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Created:
                Updated:
                Resolved: