ConcurrentModificationException if a build plan is triggered multiple times at once

XMLWordPrintable

    • Severity 3 - Minor

      Summary

      If a build plan is triggered a multiple times at once, one of the build might fail to start

      Steps to Reproduce

      Use the REST API /rest/api/latest/queue/<project>-<plan> to trigger a single builds for 7 times

      Expected Results

      The plan will start 7 builds

      Actual Results

      One of the build might fail to start. The below exception is thrown in the atlassian-bamboo.log file:

      2016-09-12 09:20:50,659 INFO [http-nio-8086-exec-19] [AccessLogFilter] klfoong POST http://localhost:8085/rest/api/latest/queue/PROJ-PLAN 1954287kb
2016-09-12 09:20:50,820 INFO [http-nio-8086-exec-125] [AccessLogFilter] klfoong POST http://localhost:8085/rest/api/latest/queue/PROJ-PLAN 1941536kb
2016-09-12 09:20:50,843 INFO [http-nio-8086-exec-24] [AccessLogFilter] klfoong POST http://localhost:8085/rest/api/latest/queue/PROJ-PLAN 1939513kb
2016-09-12 09:20:50,846 INFO [http-nio-8086-exec-33] [AccessLogFilter] klfoong POST http://localhost:8085/rest/api/latest/queue/PROJ-PLAN 1939026kb
2016-09-12 09:20:50,847 INFO [http-nio-8086-exec-50] [AccessLogFilter] klfoong POST http://localhost:8085/rest/api/latest/queue/PROJ-PLAN 1938894kb
2016-09-12 09:20:50,847 INFO [http-nio-8086-exec-124] [AccessLogFilter] klfoong POST http://localhost:8085/rest/api/latest/queue/PROJ-PLAN 1938561kb
2016-09-12 09:20:50,847 INFO [http-nio-8086-exec-1] [AccessLogFilter] klfoong POST http://localhost:8085/rest/api/latest/queue/PROJ-PLAN 1938561kb
…
2016-09-12 09:20:53,310 INFO [6-DelayedChangeDetectionThread:pool-12-thread-703] [ChainExecutionManagerImpl] Build PROJ-PLAN-JOB1-4151 has been dispatched
2016-09-12 09:20:53,311 INFO [AtlassianEvent::0-BAM::EVENTS:pool-1-thread-6] [BuildQueueManagerImpl] Attempting to queue Project - Plan - Default Job #4151 (PROJ-PLAN-JOB1-4151).
2016-09-12 09:20:53,313 INFO [AtlassianEvent::0-BAM::EVENTS:pool-1-thread-6] [ExecutionPhaseServiceImpl] Project - Plan - Default Job #4151 (PROJ-PLAN-JOB1-4151) queued
2016-09-12 09:20:53,331 ERROR [AtlassianEvent::0-BAM::EVENTS:pool-1-thread-6] [BuildContextXStreamConverter] Error context: /com.atlassian.bamboo.v2.build.BuildContextImpl/buildDefinition
2016-09-12 09:20:53,332 ERROR [AtlassianEvent::0-BAM::EVENTS:pool-1-thread-6] [BuildQueueManagerImpl] 
java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
	at java.util.HashMap$EntryIterator.next(HashMap.java:1471)
	at java.util.HashMap$EntryIterator.next(HashMap.java:1469)
	at com.atlassian.bamboo.utils.ConfigUtils.addMapToConfiguration(ConfigUtils.java:135)
	at com.atlassian.bamboo.fieldvalue.TaskConfigurationUtils.addTaskDefinitionsToConfig(TaskConfigurationUtils.java:90)
	at com.atlassian.bamboo.fieldvalue.BuildDefinitionConverter.fromObject(BuildDefinitionConverter.java:91)
	at com.atlassian.bamboo.serialization.xstream.BuildDefinitionXStreamConverter.marshal(BuildDefinitionXStreamConverter.java:48)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller$1.convertAnother(AbstractReferenceMarshaller.java:84)
	at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshallField(AbstractReflectionConverter.java:274)
	at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$2.writeField(AbstractReflectionConverter.java:250)
	at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter$2.<init>(AbstractReflectionConverter.java:213)
	at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.doMarshal(AbstractReflectionConverter.java:144)
	at com.thoughtworks.xstream.converters.reflection.AbstractReflectionConverter.marshal(AbstractReflectionConverter.java:90)
	at com.atlassian.bamboo.serialization.xstream.BuildContextXStreamConverter.marshal(BuildContextXStreamConverter.java:46)
	at com.thoughtworks.xstream.core.AbstractReferenceMarshaller.convert(AbstractReferenceMarshaller.java:69)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:58)
	at com.thoughtworks.xstream.core.TreeMarshaller.convertAnother(TreeMarshaller.java:43)
	at com.thoughtworks.xstream.core.TreeMarshaller.start(TreeMarshaller.java:82)
	at com.thoughtworks.xstream.core.AbstractTreeMarshallingStrategy.marshal(AbstractTreeMarshallingStrategy.java:37)
	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1067)
	at com.thoughtworks.xstream.XStream.marshal(XStream.java:1056)
	at com.thoughtworks.xstream.XStream.toXML(XStream.java:1029)
	at com.atlassian.bamboo.v2.build.queue.queues.XStreamQueuePersisterImpl.save(XStreamQueuePersisterImpl.java:54)
	at com.atlassian.bamboo.v2.build.queue.queues.CommonContextMaps$ForwardingOffloadingCommonContextMap.put(CommonContextMaps.java:144)
	at com.atlassian.bamboo.v2.build.queue.queues.AbstractQueueOfExecutables.enqueue(AbstractQueueOfExecutables.java:105)
	at com.atlassian.bamboo.v2.build.queue.BuildQueueManagerImpl$2.doInTransactionWithoutResult(BuildQueueManagerImpl.java:183)
	at org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:34)
	at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
	at com.atlassian.bamboo.v2.build.queue.BuildQueueManagerImpl.addToQueue(BuildQueueManagerImpl.java:175)
	at com.atlassian.bamboo.v2.build.queue.BuildTriggeredListener.handleEvent(BuildTriggeredListener.java:33)
	at sun.reflect.GeneratedMethodAccessor2355.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:36)
	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:46)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:51)
	at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:31)
	at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:20)
	at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:52)
	at java.lang.Thread.run(Thread.java:745)
2016-09-12 09:20:53,333 ERROR [AtlassianEvent::0-BAM::EVENTS:pool-1-thread-6] [AsynchronousAbleEventDispatcher] There was an exception thrown trying to dispatch event [com.atlassian.bamboo.v2.build.events.BuildTriggeredEvent[source=com.atlassian.bamboo.build.DefaultBuildExecutionManager@476c830]] from the invoker [SingleParameterMethodListenerInvoker{method=public void com.atlassian.bamboo.v2.build.queue.BuildTriggeredListener.handleEvent(com.atlassian.bamboo.v2.build.events.BuildTriggeredEvent), listener=com.atlassian.bamboo.v2.build.queue.BuildTriggeredListener@ba2bf2b}]
java.lang.RuntimeException: java.util.ConcurrentModificationException
	at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:50)
	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:46)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:51)
	at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:31)
	at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:20)
	at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:52)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
	at java.util.HashMap$EntryIterator.next(HashMap.java:1471)
	at java.util.HashMap$EntryIterator.next(HashMap.java:1469)
	at com.atlassian.bamboo.utils.ConfigUtils.addMapToConfiguration(ConfigUtils.java:135)
	at com.atlassian.bamboo.fieldvalue.TaskConfigurationUtils.addTaskDefinitionsToConfig(TaskConfigurationUtils.java:90)
...
2016-09-12 09:34:48,085 INFO [scheduler_Worker-3] [OrphanedBuildMonitorJob] PROJ-PLAN-JOB1-4151 marked as Queued but not present in queue since Mon Sep 12 09:21:48 CEST 2016
2016-09-12 09:34:48,086 ERROR [scheduler_Worker-3] [OrphanedBuildMonitorJob] Build PROJ-PLAN-JOB1-4151 had to be cancelled: it was marked as queued but was not present in the queue for (at least) the past 720 seconds.
2016-09-12 09:34:48,086 INFO [scheduler_Worker-3] [DefaultErrorHandler] Recording an error: Build PROJ-PLAN-JOB1-4151 had to be cancelled: it was marked as queued but was not present in the queue for (at least) the past 720 seconds. : PROJ-PLAN-JOB1
2016-09-12 09:34:48,089 INFO [scheduler_Worker-3] [CurrentlyBuildingContainer] removeCurrentlyBuilding called for [PROJ-PLAN-JOB1-4151]
2016-09-12 09:34:48,108 INFO [scheduler_Worker-3] [PlanStatePersisterImpl] Out of order build execution detected for build PROJ-PLAN-JOB1-4151

      Workaround

      Delay the POST REST call by a few seconds

            Assignee:
            Marcin Gardias
            Reporter:
            Foong (Inactive)
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

              Created:
              Updated:
              Resolved: