-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Low
-
Affects Version/s: 6.0.4, 6.1.1, 6.7.2, 6.7.1
-
Component/s: Builds, Performance
-
4
-
Severity 1 - Critical
-
1
Summary
When a build with huge number of files modified in a single commit is re-run, Bamboo can no longer build any other plan.
Steps to Reproduce
- Use any repository, e.g. Git
- Create a plan and configure it to checkout that repository
- Run the plan
- Commit 100k file changes to the repository
- Run the plan again
- Attempt to re-run the plan
Expected Results
Bamboo re-runs the plan.
Actual Results
restartBuild.action request is non-performant and runs for an extended period (depending on number of commit files):
18-Nov-2017 10:25:42.884 WARNING [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread "http-nio-8085-exec-85" (id=375) has been active for 69,972 milliseconds (since 11/18/17 10:24 AM) to serve the same request for https://bamboo/build/admin/restartBuild.action?planKey=PLAN-PROJ&buildNumber=2 and may be stuck (configured threshold for this StuckThreadDetectionValve is 60 seconds). There is/are 3 thread(s) in total that are monitored by this Valve and may be stuck. java.lang.Throwable at org.hibernate.internal.util.compare.EqualsHelper.equals(EqualsHelper.java:31) at org.hibernate.type.descriptor.java.AbstractTypeDescriptor.areEqual(AbstractTypeDescriptor.java:70) at org.hibernate.type.AbstractStandardBasicType.isEqual(AbstractStandardBasicType.java:197) at org.hibernate.type.AbstractStandardBasicType.isSame(AbstractStandardBasicType.java:185) at org.hibernate.type.ComponentType.isSame(ComponentType.java:168) at org.hibernate.collection.internal.PersistentBag.countOccurrences(PersistentBag.java:133) at org.hibernate.collection.internal.PersistentBag.equalsSnapshot(PersistentBag.java:115) at org.hibernate.engine.spi.CollectionEntry.dirty(CollectionEntry.java:158) at org.hibernate.engine.spi.CollectionEntry.preFlush(CollectionEntry.java:181) at org.hibernate.event.internal.AbstractFlushingEventListener.prepareCollectionFlushes(AbstractFlushingEventListener.java:178) at org.hibernate.event.internal.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:78) at org.hibernate.event.internal.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:44) at org.hibernate.internal.SessionImpl.autoFlushIfRequired(SessionImpl.java:1396) at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1889) at org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:366) at org.hibernate.internal.CriteriaImpl.uniqueResult(CriteriaImpl.java:388) at com.atlassian.bamboo.resultsummary.BuildResultsSummaryHibernateDao$2.doInHibernate(BuildResultsSummaryHibernateDao.java:156) at com.atlassian.bamboo.resultsummary.BuildResultsSummaryHibernateDao$2.doInHibernate(BuildResultsSummaryHibernateDao.java:148) at org.springframework.orm.hibernate5.HibernateTemplate.doExecute(HibernateTemplate.java:359) at org.springframework.orm.hibernate5.HibernateTemplate.execute(HibernateTemplate.java:313) at com.atlassian.bamboo.resultsummary.BuildResultsSummaryHibernateDao.getResultsSummary(BuildResultsSummaryHibernateDao.java:148) at com.atlassian.bamboo.resultsummary.BuildResultsSummaryManagerImpl.getResultsSummary(BuildResultsSummaryManagerImpl.java:182) at com.atlassian.bamboo.resultsummary.BuildResultsSummaryManagerImpl.getResultsSummary(BuildResultsSummaryManagerImpl.java:176) at sun.reflect.GeneratedMethodAccessor1553.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.acegisecurity.intercept.method.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:66) at com.atlassian.bamboo.security.acegi.intercept.aopalliance.AuthorityOverrideMethodSecurityInterceptor.invoke(AuthorityOverrideMethodSecurityInterceptor.java:30) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy100.getResultsSummary(Unknown Source) at com.atlassian.bamboo.resultsummary.variables.ResultsSummaryVariableAccessorImpl.getValidChainResultSummaryByKey(ResultsSummaryVariableAccessorImpl.java:53) at com.atlassian.bamboo.resultsummary.variables.ResultsSummaryVariableAccessorImpl.calculateCurrentVariablesState(ResultsSummaryVariableAccessorImpl.java:277) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213) at com.sun.proxy.$Proxy127.calculateCurrentVariablesState(Unknown Source) at com.atlassian.bamboo.v2.trigger.ContinuedBuildDetectionAction.generateResultWithoutChanges(ContinuedBuildDetectionAction.java:177)
The plan is not re-run and blocks all other plans from being run. Stays so until Bamboo is restarted or connection to the DB dies, possibly after hours.
Notes
The problem seems to be related to Hibernate and the way it's checking dirty state of entities. Commit files are implemented as 'bag' in the commit entity, and dirty check for a 'bag' pessimistically requires N^2 operations. For 100k sized bag it would take 10B comparisons to determine dirtiness. Bamboo most likely acquires some lock to perform the calculation, causing starvation to the build queue, and never actually executing the build itself.
Workaround
Avoid re-running builds with hundreds of thousands of files changed per commit.