Details
-
Bug
-
Resolution: Unresolved
-
Low
-
None
-
6.7.1
-
26
-
Severity 2 - Major
-
9
-
Description
Summary
Quiet period setting on a a repository will sleep an entire PlanExec thread for the duration of the quiet period and subsequent retries. This can easily lead to thread starvation for the limited thread pool of 4 with long and frequent quiet period usage – leading to triggered build and deployment delays even for plans that do not have the quiet period configured (because they must pass through the queue serviced by the same worker pool)
Steps to Reproduce
- Set a quiet period delay of 600 seconds on 4 separate plans referencing the same repository.
- Add a repository polling trigger.
- Create another plan with a repository polling trigger referencing a different repository.
- Commit to the repository and wait for polling trigger.
- All 4 plans will take up a PlanExec thread for 600 seconds.
- Commit to the second repository to attempt to trigger the other plan.
- Build will remain queued until PlanExec threads clear up from the 600 second quiet period.
Expected Results
Quiet period should not block change detection for other plans.
Actual Results
Change detection is blocked. Example stack for sleeping thread:
"9-BAM::PlanExec:pool-15-thread-2" #167 daemon prio=1 os_prio=0 tid=0x00007eff74f9e800 nid=0x14ac waiting on condition [0x00007eff164c1000] java.lang.Thread.State: TIMED_WAITING (sleeping) at java.lang.Thread.sleep(Native Method) at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectChangesAfterQuietPeriod(DefaultChangeDetectionManager.java:562) at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.lambda$createBuildRepositoryChanges$2(DefaultChangeDetectionManager.java:470) at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager$$Lambda$508/252415022.call(Unknown Source) at com.atlassian.bamboo.variable.CustomVariableContextImpl.withVariableSubstitutor(CustomVariableContextImpl.java:185) at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectChangesSinceLastBuildInternal(DefaultChangeDetectionManager.java:440) at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectChangesSinceRevisions(DefaultChangeDetectionManager.java:290) at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectChangesSinceRevisions(DefaultChangeDetectionManager.java:215) at com.atlassian.bamboo.v2.trigger.DefaultChangeDetectionManager.collectChangesSinceLastBuildIfTriggered(DefaultChangeDetectionManager.java:137) at com.atlassian.bamboo.v2.trigger.ChangeDetectionListenerAction.testIfBuildShouldStart(ChangeDetectionListenerAction.java:105) at com.atlassian.bamboo.plan.PlanExecutionManagerImpl$3.call(PlanExecutionManagerImpl.java:448) at com.atlassian.bamboo.plan.PlanExecutionManagerImpl$3.call(PlanExecutionManagerImpl.java:435) at io.atlassian.util.concurrent.ManagedLocks$ManagedLockImpl.withLock(ManagedLocks.java:293) at com.atlassian.bamboo.plan.PlanExecutionLockServiceImpl.lock(PlanExecutionLockServiceImpl.java:75) at com.atlassian.bamboo.plan.PlanExecutionManagerImpl.doWithProcessLock(PlanExecutionManagerImpl.java:655) at com.atlassian.bamboo.plan.PlanExecutionManagerImpl.startConditionalBuild(PlanExecutionManagerImpl.java:435) at com.atlassian.bamboo.plan.PlanExecutionManagerImpl.start(PlanExecutionManagerImpl.java:487) at com.atlassian.bamboo.plan.PlanExecutionManagerImpl.start(PlanExecutionManagerImpl.java:500) at com.atlassian.bamboo.plan.DelegatingPlanExecutionManager.start(DelegatingPlanExecutionManager.java:79) at com.atlassian.bamboo.plan.NonBlockingPlanExecutionServiceImpl$4.startPlanExecution(NonBlockingPlanExecutionServiceImpl.java:197) at com.atlassian.bamboo.plan.NonBlockingPlanExecutionServiceImpl$4.lambda$call$0(NonBlockingPlanExecutionServiceImpl.java:186) at com.atlassian.bamboo.plan.NonBlockingPlanExecutionServiceImpl$4$$Lambda$453/1303392598.run(Unknown Source) at com.atlassian.bamboo.util.CacheAwareness$3.call(CacheAwareness.java:136) at com.atlassian.bamboo.util.CacheAwareness$3.call(CacheAwareness.java:133) at com.atlassian.bamboo.util.CacheAwareness.withValuesOlderThanTimestampReloaded(CacheAwareness.java:162) at com.atlassian.bamboo.util.CacheAwareness.withValuesOlderThanTimestampReloaded(CacheAwareness.java:133) at com.atlassian.bamboo.util.CacheAwareness.withValuesOlderThanTimestampReloaded(CacheAwareness.java:187) at com.atlassian.bamboo.plan.NonBlockingPlanExecutionServiceImpl$4.call(NonBlockingPlanExecutionServiceImpl.java:185) at com.atlassian.bamboo.plan.NonBlockingPlanExecutionServiceImpl$4.call(NonBlockingPlanExecutionServiceImpl.java:170) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:48) at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:26) at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:17) at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:41) at java.lang.Thread.run(Thread.java:748)
Workaround
Reduce or disable quiet period usage across the instance. Limit it to seconds rather than minutes and reduce retries:
- Visit each plan using Quiet Period enabled repositories and either disable the Quiet Period setting or lower it : Plan Configuration >> Repositories >> repository >> Change Detection Options >> Enable Quiet Period
- Less retries is also better.
This is a workaround designed to mitigate the impact of the contention but without addressing the Quiet Period usage, the problem will almost certainly still occur at some level:
- Edit your <bamboo-install>/bin/setenv.sh
- Add the following -Dbamboo.plan.exe.threads=8 to your JVM_SUPPORT_RECOMMENDED_ARGS. E.g.
# # Occasionally Atlassian Support may recommend that you set some specific JVM arguments. You can use this variable below to do that. # : ${JVM_SUPPORT_RECOMMENDED_ARGS:="-Dbamboo.plan.exe.threads=8"}
- Restart Bamboo
Attachments
Issue Links
- is related to
-
-
- Gathering Impact
-
- supersedes
-
BAM-18638 Quiet Period Validation
- Gathering Interest
- mentioned in
-
Page Loading...
-
-
-
-
-
-
-
-
-
- relates to
-