Uploaded image for project: 'Bamboo Data Center'
  1. Bamboo Data Center
  2. BAM-17890

ConcurrentModificationException if a build plan is triggered multiple times at once

    XMLWordPrintable

Details

    Description

      Summary

      If a build plan is triggered a multiple times at once, one of the build might fail to start

      Steps to Reproduce

      Use the REST API /rest/api/latest/queue/<project>-<plan> to trigger a single builds for 7 times

      Expected Results

      The plan will start 7 builds

      Actual Results

      One of the build might fail to start. The below exception is thrown in the atlassian-bamboo.log file:

      2016-09-12 09:20:50,659 INFO [http-nio-8086-exec-19] [AccessLogFilter] klfoong POST http://localhost:8085/rest/api/latest/queue/PROJ-PLAN 1954287kb
      2016-09-12 09:20:50,820 INFO [http-nio-8086-exec-125] [AccessLogFilter] klfoong POST http://localhost:8085/rest/api/latest/queue/PROJ-PLAN 1941536kb
      2016-09-12 09:20:50,843 INFO [http-nio-8086-exec-24] [AccessLogFilter] klfoong POST http://localhost:8085/rest/api/latest/queue/PROJ-PLAN 1939513kb
      2016-09-12 09:20:50,846 INFO [http-nio-8086-exec-33] [AccessLogFilter] klfoong POST http://localhost:8085/rest/api/latest/queue/PROJ-PLAN 1939026kb
      2016-09-12 09:20:50,847 INFO [http-nio-8086-exec-50] [AccessLogFilter] klfoong POST http://localhost:8085/rest/api/latest/queue/PROJ-PLAN 1938894kb
      2016-09-12 09:20:50,847 INFO [http-nio-8086-exec-124] [AccessLogFilter] klfoong POST http://localhost:8085/rest/api/latest/queue/PROJ-PLAN 1938561kb
      2016-09-12 09:20:50,847 INFO [http-nio-8086-exec-1] [AccessLogFilter] klfoong POST http://localhost:8085/rest/api/latest/queue/PROJ-PLAN 1938561kb
      …
      2016-09-12 09:20:53,310 INFO [6-DelayedChangeDetectionThread:pool-12-thread-703] [ChainExecutionManagerImpl] Build PROJ-PLAN-JOB1-4151 has been dispatched
      2016-09-12 09:20:53,311 INFO [AtlassianEvent::0-BAM::EVENTS:pool-1-thread-6] [BuildQueueManagerImpl] Attempting to queue Project - Plan - Default Job #4151 (PROJ-PLAN-JOB1-4151).
      2016-09-12 09:20:53,313 INFO [AtlassianEvent::0-BAM::EVENTS:pool-1-thread-6] [ExecutionPhaseServiceImpl] Project - Plan - Default Job #4151 (PROJ-PLAN-JOB1-4151) queued
      2016-09-12 09:20:53,331 ERROR [AtlassianEvent::0-BAM::EVENTS:pool-1-thread-6] [BuildContextXStreamConverter] Error context: /com.atlassian.bamboo.v2.build.BuildContextImpl/buildDefinition
      2016-09-12 09:20:53,332 ERROR [AtlassianEvent::0-BAM::EVENTS:pool-1-thread-6] [BuildQueueManagerImpl] 
      java.util.ConcurrentModificationException
      	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
      	at java.util.HashMap$EntryIterator.next(HashMap.java:1471)
      	at java.util.HashMap$EntryIterator.next(HashMap.java:1469)
      	at com.atlassian.bamboo.utils.ConfigUtils.addMapToConfiguration(ConfigUtils.java:135)
      	at com.atlassian.bamboo.fieldvalue.TaskConfigurationUtils.addTaskDefinitionsToConfig(TaskConfigurationUtils.java:90)
      	at com.atlassian.bamboo.fieldvalue.BuildDefinitionConverter.fromObject(BuildDefinitionConverter.java:91)
      	at com.atlassian.bamboo.serialization.xstream.BuildDefinitionXStreamConverter.marshal(BuildDefinitionXStreamConverter.java:48)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
      	at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshallField(AbstractReflectionConverter.java:274)
      	at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$2.writeField(AbstractReflectionConverter.java:250)
      	at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$2.<init>(AbstractReflectionConverter.java:213)
      	at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doMarshal(AbstractReflectionConverter.java:144)
      	at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshal(AbstractReflectionConverter.java:90)
      	at com.atlassian.bamboo.serialization.xstream.BuildContextXStreamConverter.marshal(BuildContextXStreamConverter.java:46)
      	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
      	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
      	at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
      	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
      	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1067)
      	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1056)
      	at com.thoughtworks.xstream.XStream.toXML(XStream.java:1029)
      	at com.atlassian.bamboo.v2.build.queue.queues.XStreamQueuePersisterImpl.save(XStreamQueuePersisterImpl.java:54)
      	at com.atlassian.bamboo.v2.build.queue.queues.CommonContextMaps$ForwardingOffloadingCommonContextMap.put(CommonContextMaps.java:144)
      	at com.atlassian.bamboo.v2.build.queue.queues.AbstractQueueOfExecutables.enqueue(AbstractQueueOfExecutables.java:105)
      	at com.atlassian.bamboo.v2.build.queue.BuildQueueManagerImpl$2.doInTransactionWithoutResult(BuildQueueManagerImpl.java:183)
      	at org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:34)
      	at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
      	at com.atlassian.bamboo.v2.build.queue.BuildQueueManagerImpl.addToQueue(BuildQueueManagerImpl.java:175)
      	at com.atlassian.bamboo.v2.build.queue.BuildTriggeredListener.handleEvent(BuildTriggeredListener.java:33)
      	at sun.reflect.GeneratedMethodAccessor2355.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:36)
      	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:46)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:51)
      	at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:31)
      	at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:20)
      	at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:52)
      	at java.lang.Thread.run(Thread.java:745)
      2016-09-12 09:20:53,333 ERROR [AtlassianEvent::0-BAM::EVENTS:pool-1-thread-6] [AsynchronousAbleEventDispatcher] There was an exception thrown trying to dispatch event [com.atlassian.bamboo.v2.build.events.BuildTriggeredEvent[source=com.atlassian.bamboo.build.DefaultBuildExecutionManager@476c830]] from the invoker [SingleParameterMethodListenerInvoker{method=public void com.atlassian.bamboo.v2.build.queue.BuildTriggeredListener.handleEvent(com.atlassian.bamboo.v2.build.events.BuildTriggeredEvent), listener=com.atlassian.bamboo.v2.build.queue.BuildTriggeredListener@ba2bf2b}]
      java.lang.RuntimeException: java.util.ConcurrentModificationException
      	at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:50)
      	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:46)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:51)
      	at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:31)
      	at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:20)
      	at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:52)
      	at java.lang.Thread.run(Thread.java:745)
      Caused by: java.util.ConcurrentModificationException
      	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
      	at java.util.HashMap$EntryIterator.next(HashMap.java:1471)
      	at java.util.HashMap$EntryIterator.next(HashMap.java:1469)
      	at com.atlassian.bamboo.utils.ConfigUtils.addMapToConfiguration(ConfigUtils.java:135)
      	at com.atlassian.bamboo.fieldvalue.TaskConfigurationUtils.addTaskDefinitionsToConfig(TaskConfigurationUtils.java:90)
      ...
      2016-09-12 09:34:48,085 INFO [scheduler_Worker-3] [OrphanedBuildMonitorJob] PROJ-PLAN-JOB1-4151 marked as Queued but not present in queue since Mon Sep 12 09:21:48 CEST 2016
      2016-09-12 09:34:48,086 ERROR [scheduler_Worker-3] [OrphanedBuildMonitorJob] Build PROJ-PLAN-JOB1-4151 had to be cancelled: it was marked as queued but was not present in the queue for (at least) the past 720 seconds.
      2016-09-12 09:34:48,086 INFO [scheduler_Worker-3] [DefaultErrorHandler] Recording an error: Build PROJ-PLAN-JOB1-4151 had to be cancelled: it was marked as queued but was not present in the queue for (at least) the past 720 seconds. : PROJ-PLAN-JOB1
      2016-09-12 09:34:48,089 INFO [scheduler_Worker-3] [CurrentlyBuildingContainer] removeCurrentlyBuilding called for [PROJ-PLAN-JOB1-4151]
      2016-09-12 09:34:48,108 INFO [scheduler_Worker-3] [PlanStatePersisterImpl] Out of order build execution detected for build PROJ-PLAN-JOB1-4151
      

      Workaround

      Delay the POST REST call by a few seconds

      Attachments

        Issue Links

          Activity

            People

              mgardias Marcin Gardias
              klfoong Foong (Inactive)
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: