Deadlock issue while view currently running build

XMLWordPrintable

    • Type: Bug
    • Resolution: Fixed
    • Priority: Medium
    • 1.2
    • Affects Version/s: 1.1.2
    • Component/s: Builds, Infrastructure
    • None

      There's a deadlock in the getCurrentlyRunningBuild since we use the LinkedBlockingQueue as a lock (bad idea). Thd deadlock looks a little like

      Found one Java-level deadlock:
      =============================
      "TP-Processor34":
        waiting to lock monitor 0x08d3f500 (object 0x55f28d48, a edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock),
        which is held by "BAM:queuedTaskMaster:BuildExe"
      "BAM:queuedTaskMaster:BuildExe":
        waiting to lock monitor 0x08d3f3cc (object 0x55f22868, a edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue),
        which is held by "TP-Processor34"
      
      Java stack information for the threads listed above:
      ===================================================
      "TP-Processor34":
      	at edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$Itr. (LinkedBlockingQueue.java:646)
      	- waiting to lock 0x55f28d48> (a edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
      	- locked 0x55f28d50> (a edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
      	at edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue.iterator(LinkedBlockingQueue.java:630)
      	at com.atlassian.bamboo.build.pipeline.tasks.BambooBuildPipeline.getBuildResultForBuild(BambooBuildPipeline.java:318)
      	- locked 0x55f22868> (a edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue)
      	at com.atlassian.bamboo.build.HibernateBuildManager.getCurrentBuildingResult(HibernateBuildManager.java:350)
      	at sun.reflect.GeneratedMethodAccessor201.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:585)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:287)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:181)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:148)
      	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:170)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:176)
      	at $Proxy75.getCurrentBuildingResult(Unknown Source)
      	at com.atlassian.bamboo.build.ViewBuild.getCurrentlyRunningBuild(ViewBuild.java:59)
      	at sun.reflect.GeneratedMethodAccessor3194.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      	at java.lang.reflect.Method.invoke(Method.java:585)
      	at ognl.OgnlRuntime.invokeMethod(OgnlRuntime.java:491)
      	at ognl.OgnlRuntime.getMethodValue(OgnlRuntime.java:904)
      	at ognl.ObjectPropertyAccessor.getPossibleProperty(ObjectPropertyAccessor.java:54)
      	at ognl.ObjectPropertyAccessor.getProperty(ObjectPropertyAccessor.java:122)
      	at com.opensymphony.xwork.util.OgnlValueStack$ObjectAccessor.getProperty(OgnlValueStack.java:53)
      	at ognl.OgnlRuntime.getProperty(OgnlRuntime.java:1616)
      	at com.opensymphony.xwork.util.CompoundRootAccessor.getProperty(CompoundRootAccessor.java:102)
      	at ognl.OgnlRuntime.getProperty(OgnlRuntime.java:1616)
      	at ognl.ASTProperty.getValueBody(ASTProperty.java:96)
      	at ognl.SimpleNode.evaluateGetValueBody(SimpleNode.java:170)
      	at ognl.SimpleNode.getValue(SimpleNode.java:210)
      	at ognl.Ognl.getValue(Ognl.java:333)
      	at ognl.Ognl.getValue(Ognl.java:310)
      	at com.opensymphony.xwork.util.OgnlUtil.getValue(OgnlUtil.java:188)
      	at com.opensymphony.xwork.util.OgnlValueStack.findValue(OgnlValueStack.java:205)
      	at com.opensymphony.webwork.views.freemarker.ScopesHashModel.get(ScopesHashModel.java:51)
      	at freemarker.core.Environment.getGlobalVariable(Environment.java:976)
      	at freemarker.core.Environment.getVariable(Environment.java:962)
      	at freemarker.core.Identifier._getAsTemplateModel(Identifier.java:70)
      	at freemarker.core.Expression.getAsTemplateModel(Expression.java:89)
      	at freemarker.core.BuiltIn$existsBI._getAsTemplateModel(BuiltIn.java:635)
      	at freemarker.core.BuiltIn$existsBI.isTrue(BuiltIn.java:646)
      	at freemarker.core.IfBlock.accept(IfBlock.java:80)
      	at freemarker.core.Environment.visit(Environment.java:196)
      	at freemarker.core.MixedContent.accept(MixedContent.java:92)
      	at freemarker.core.Environment.visit(Environment.java:196)
      	at freemarker.core.ConditionalBlock.accept(ConditionalBlock.java:79)
      	at freemarker.core.Environment.visit(Environment.java:196)
      	at freemarker.core.Environment.visit(Environment.java:233)
      	at freemarker.core.UnifiedCall.accept(UnifiedCall.java:116)
      	at freemarker.core.Environment.visit(Environment.java:196)
      	at freemarker.core.Environment.process(Environment.java:176)
      	at freemarker.template.Template.process(Template.java:232)
      	at com.opensymphony.webwork.views.freemarker.FreemarkerResult.doExecute(FreemarkerResult.java:130)
      	at com.opensymphony.webwork.dispatcher.WebWorkResultSupport.execute(WebWorkResultSupport.java:101)
      	at com.opensymphony.xwork.DefaultActionInvocation.executeResult(DefaultActionInvocation.java:312)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:207)
      	at com.opensymphony.xwork.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:137)
      	at com.atlassian.bamboo.ww2.interceptors.BambooWorkflowInterceptor.doIntercept(BambooWorkflowInterceptor.java:33)
      	at com.opensymphony.xwork.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:81)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:189)
      	at com.opensymphony.xwork.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:115)
      	at com.atlassian.bamboo.ww2.interceptors.BambooValidationInterceptor.doIntercept(BambooValidationInterceptor.java:33)
      	at com.opensymphony.xwork.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:81)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:189)
      	at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:189)
      	at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:189)
      	at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:189)
      	at com.opensymphony.webwork.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:171)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:189)
      	at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:189)
      	at com.atlassian.bamboo.ww2.interceptors.BasePermissionCheckInterceptor.intercept(BasePermissionCheckInterceptor.java:38)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:189)
      	at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:189)
      	at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:189)
      	at com.opensymphony.xwork.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:151)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:189)
      	at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:189)
      	at com.atlassian.bamboo.ww2.interceptors.PaginationAwareInterceptor.intercept(PaginationAwareInterceptor.java:68)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:189)
      	at com.atlassian.bamboo.ww2.interceptors.StatisticsAwareInterceptor.intercept(StatisticsAwareInterceptor.java:42)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:189)
      	at com.atlassian.bamboo.ww2.interceptors.ResultsListAwareInterceptor.intercept(ResultsListAwareInterceptor.java:43)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:189)
      	at com.atlassian.bamboo.ww2.interceptors.TestCaseAwareInterceptor.intercept(TestCaseAwareInterceptor.java:43)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:189)
      	at com.atlassian.bamboo.ww2.interceptors.BuildAwareInterceptor.intercept(BuildAwareInterceptor.java:41)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:189)
      	at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:189)
      	at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:31)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:189)
      	at com.opensymphony.xwork.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:100)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:189)
      	at com.opensymphony.xwork.DefaultActionProxy.execute(DefaultActionProxy.java:113)
      	at com.opensymphony.webwork.dispatcher.DispatcherUtils.serviceAction(DispatcherUtils.java:233)
      	at com.opensymphony.webwork.dispatcher.FilterDispatcher.doFilter(FilterDispatcher.java:198)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      	at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:118)
      	at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:52)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      	at com.opensymphony.webwork.dispatcher.ActionContextCleanUp.doFilter(ActionContextCleanUp.java:78)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      	at com.atlassian.bamboo.filter.AccessLogFilter.doFilter(AccessLogFilter.java:85)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:264)
      	at org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:107)
      	at org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)
      	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
      	at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:110)
      	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
      	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
      	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
      	at com.atlassian.bamboo.filter.SeraphLoginFilter.doFilter(SeraphLoginFilter.java:56)
      	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
      	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:191)
      	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:274)
      	at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:148)
      	at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:90)
      	at com.atlassian.bamboo.filter.BambooAcegiProxyFilter.doFilter(BambooAcegiProxyFilter.java:25)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      	at com.atlassian.bamboo.filter.LicenseFilter.doFilter(LicenseFilter.java:71)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      	at com.atlassian.johnson.filters.JohnsonFilter.doFilter(JohnsonFilter.java:91)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      	at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:182)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      	at com.atlassian.seraph.filter.LoginFilter.doFilter(LoginFilter.java:177)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      	at org.springframework.orm.hibernate.support.OpenSessionInViewFilter.doFilterInternal(OpenSessionInViewFilter.java:174)
      	at com.atlassian.spring.filter.FlushingSpringSessionInViewFilter.doFilterInternal(FlushingSpringSessionInViewFilter.java:29)
      	at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:77)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      	at com.atlassian.bamboo.filter.CompressingFilter.doFilter(CompressingFilter.java:69)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      	at com.atlassian.bamboo.filter.NonCachingHeaderFilter.doFilter(NonCachingHeaderFilter.java:22)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      	at com.atlassian.bamboo.filter.RequestCacheThreadLocalFilter.doFilter(RequestCacheThreadLocalFilter.java:26)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:202)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:173)
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:213)
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:178)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
      	at org.jstripe.tomcat.probe.Tomcat55AgentValve.invoke(Tomcat55AgentValve.java:20)
      	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
      	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
      	at org.apache.jk.server.JkCoyoteHandler.invoke(JkCoyoteHandler.java:199)
      	at org.apache.jk.common.HandlerRequest.invoke(HandlerRequest.java:282)
      	at org.apache.jk.common.ChannelSocket.invoke(ChannelSocket.java:767)
      	at org.apache.jk.common.ChannelSocket.processConnection(ChannelSocket.java:697)
      	at org.apache.jk.common.ChannelSocket$SocketConnection.runIt(ChannelSocket.java:889)
      	at org.apache.tomcat.util.threads.ThreadPool$ControlRunnable.run(ThreadPool.java:684)
      	at java.lang.Thread.run(Thread.java:595)
      "BAM:queuedTaskMaster:BuildExe":
      	at edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:323)
      	- waiting to lock 0x55f22868> (a edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue)
      	- locked 0x55f28d48> (a edu.emory.mathcs.backport.java.util.concurrent.LinkedBlockingQueue$SerializableLock)
      	at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:921)
      	at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:980)
      	at edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:528)
      	at edu.emory.mathcs.backport.java.util.concurrent.helpers.ThreadHelpers$1.run(ThreadHelpers.java:34)
      	at java.lang.Thread.run(Thread.java:595)
      

        1. catalina.out
          1.97 MB
          MarkC

              Assignee:
              MarkC
              Reporter:
              MarkC
              Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

                Created:
                Updated:
                Resolved:

                  Estimated:
                  Original Estimate - 3h
                  3h
                  Remaining:
                  Remaining Estimate - 3h
                  3h
                  Logged:
                  Time Spent - Not Specified
                  Not Specified