Uploaded image for project: 'Bamboo Data Center'
  1. Bamboo Data Center
  2. BAM-21873

Elastic agents are continuously started due to a failed build in queue

    XMLWordPrintable

Details

    • Bug
    • Resolution: Fixed
    • High
    • 9.1.0, 8.2.7, 9.0.2
    • 8.0.4, 8.2.3, 8.2.6
    • Elastic Bamboo
    • None

    Description

      Issue Summary

      Pending jobs should be stopped when removed from specs, but due to some conditions which is not presently reproducible, the jobs are not stopped.

      If Automatic Elastic agent management is enabled, Bamboo will spin up a new elastic agent each time it tries to run the jobs, and these agents will remain in pending state. This will continue until the Allowed non-Bamboo instances is reached.

      The error is seen in the logs

      2022-07-27 08:47:53,802 INFO [AtlassianEvent::0-BAM::EVENTS:pool-1-thread-58] [XStreamQueuePersisterImpl] Loading offloaded context for: CB-BUILD-JOB1-1
      2022-07-27 08:47:53,807 ERROR [AtlassianEvent::0-BAM::EVENTS:pool-1-thread-58] [AsynchronousAbleEventDispatcher] There was an exception thrown trying to dispatch event [AgentConfigurationUpdatedEvent Elastic Agent on i-03017056f7504e27a] from the invoker [com.atlassian.event.internal.ComparableListenerInvoker@4c7938f8]
      java.lang.RuntimeException: Plan CB-BUILD-JOB1-1 is not a DB object. Listener: com.atlassian.bamboo.v2.build.queue.BuildQueueUpdateListener event: com.atlassian.bamboo.event.AgentConfigurationUpdatedEvent
      	at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:53)
      	at com.atlassian.event.internal.ComparableListenerInvoker.invoke(ComparableListenerInvoker.java:48)
      	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.lambda$null$0(AsynchronousAbleEventDispatcher.java:37)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      	at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:48)
      	at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:26)
      	at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:17)
      	at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:41)
      	at java.lang.Thread.run(Thread.java:748)
      Caused by: java.lang.IllegalStateException: Plan CB-BUILD-JOB1-1 is not a DB object
      	at com.atlassian.bamboo.plan.cache.AbstractImmutablePlan.getId(AbstractImmutablePlan.java:122)
      	at com.atlassian.bamboo.utils.BambooPredicates$HasBambooObjectEqualIdPredicate.apply(BambooPredicates.java:176)
      	at com.atlassian.bamboo.utils.BambooPredicates$HasBambooObjectEqualIdPredicate.apply(BambooPredicates.java:167)
      	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
      	at java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958)
      	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
      	at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499)
      	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486)

       

      This is reproducible on Data Center: N/A - not reproducible, but it occurs on both DC and Server

       

      Steps to Reproduce

      N/A - There is no step to reproduce it at the moment

      Expected Results

      N/A

      Actual Results

      N/A

      Workaround

      Since the use of SQL update is not an advisable way to alter the job state,  a recommended options is to limit the number of elastic agent created by Bamboo are;

      1. disable automatic agent management
      2. or use custom elastic agent management

       

      Attachments

        Issue Links

          Activity

            People

              mgardias Marcin Gardias
              voseghale VICTOR-OSEGHALE
              Votes:
              1 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: