Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-39393

Confluence performance can degrade when JIRA/Bamboo has performance problems due to gadget feed retrieval

    XMLWordPrintable

Details

    Description

      It was observed when Confluence is configured to retrieve a gadget feed from an external application, such as JIRA or Bamboo, if that application becomes unresponsive, Confluence can also gradually become unresponsive as well. This is due to threads attempting to read gadget feeds from the linked application, triggered by common Confluence Editor operations such as inserting a macro or loading the macro browser.

      Thread dumps taken from the Confluence JVM during this time will show the majority of HTTP threads (245 out of 300 in the case observed) looking like the below:

      "http-nio-8090-exec-232" #424 daemon prio=5 os_prio=0 tid=0x00007f4aac0d0000 nid=0xca62 waiting on condition [0x00007f47c49c4000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000003cd4014d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
      	at org.codehaus.httpcache4j.cache.Mutex.acquire(Mutex.java:45)
      	at org.codehaus.httpcache4j.cache.HTTPCache.doCachedRequest(HTTPCache.java:103)
      	at org.codehaus.httpcache4j.cache.HTTPCache.doCachedRequest(HTTPCache.java:80)
      	at com.atlassian.gadgets.directory.internal.impl.GadgetFeedReaderImpl.<init>(GadgetFeedReaderImpl.java:46)
      	at com.atlassian.gadgets.directory.internal.impl.GadgetFeedReaderFactoryImpl.getFeedReader(GadgetFeedReaderFactoryImpl.java:28)
      	at sun.reflect.GeneratedMethodAccessor3143.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:497)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307)
      	at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:58)
      	at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:62)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      	at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:56)
      	at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:39)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      	at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
      	at com.sun.proxy.$Proxy2822.getFeedReader(Unknown Source)
      	at com.atlassian.confluence.plugins.gadgets.metadata.GadgetsMacroMetadataProvider.getUrisFromFeedStore(GadgetsMacroMetadataProvider.java:240)
      	at com.atlassian.confluence.plugins.gadgets.metadata.GadgetsMacroMetadataProvider.getGadgetUris(GadgetsMacroMetadataProvider.java:225)
      	at com.atlassian.confluence.plugins.gadgets.metadata.CachingGadgetsMacroMetadataProvider.getGadgetUris(CachingGadgetsMacroMetadataProvider.java:181)
      	at com.atlassian.confluence.plugins.gadgets.metadata.CachingGadgetsMacroMetadataProvider.getData(CachingGadgetsMacroMetadataProvider.java:116)
      	at com.atlassian.confluence.plugins.gadgets.metadata.CachingGadgetsMacroMetadataProvider.getSummaries(CachingGadgetsMacroMetadataProvider.java:157)
      	at com.atlassian.confluence.macro.browser.DefaultMacroMetadataManager.buildMacroSummaries(DefaultMacroMetadataManager.java:123)
      	at com.atlassian.confluence.macro.browser.DefaultMacroMetadataManager.getAllMacroSummaries(DefaultMacroMetadataManager.java:42)
      	at com.atlassian.confluence.macro.browser.DefaultMacroBrowserManager.getMacroSummaries(DefaultMacroBrowserManager.java:66)
      	at com.atlassian.confluence.macro.browser.actions.BrowseMacrosAction.getSummaries(BrowseMacrosAction.java:58)
      	at com.atlassian.confluence.macro.browser.actions.BrowseMacrosAction.execute(BrowseMacrosAction.java:36)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:168)
      	at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:35)
      	at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165)
      	at com.atlassian.confluence.core.ConfluenceWorkflowInterceptor.intercept(ConfluenceWorkflowInterceptor.java:65)
      ...
      	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
      	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
      	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
      	at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)
      	at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
      	at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)
      	at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950)
      	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)
      	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
      	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
      	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
      	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040)
      	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)
      	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1721)
      	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1679)
      	- locked <0x000000048d4da578> (a org.apache.tomcat.util.net.NioChannel)
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      	at java.lang.Thread.run(Thread.java:745)
      

      These threads will be long-running and appear in multiple thread dumps taken over a period of time. The impact to users is that eventually Confluence will not have enough threads free in order to service users.

      Other indications include in atlassian-confluence.log messages like

      atlassian-confluence.log
      2017-08-14 11:45:46,664 ERROR [http-nio-8887-exec-6] [[Standalone].[localhost].[/].[servlet-module-container-servlet]] log Servlet.service() for servlet [servlet-module-container-servlet] in context with path [] threw exception
      java.lang.RuntimeException: java.util.concurrent.TimeoutException
      

      and

      atlassian-confluence.log
      2017-08-14 11:55:32,685 WARN [http-nio-8887-exec-10] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1502682314843, threadId=146, threadName='http-nio-8887-exec-10', userId='<unknown>', type='web-request', summary='/plugins/macrobrowser/browse-macros.action?detailed=false&macroMetadataClientCacheKey=1502682297826'}
       -- referer: https://confluence.mycompany.com/display/SK/This+is+a+page | url: /plugins/macrobrowser/browse-macros.action | traceId: 8bfdb50a52b72392 | userName: userXYZ
      

      Workaround

      Effort should be made to address the JIRA or Bamboo unavailability problem as that is the trigger for this bug. In the short term, the offending gadget feed may be temporarily removed from Confluence Admin > External Gadgets in order to avoid further outages.

      Attachments

        Issue Links

          Activity

            People

              zzhou Zhenhuan Zhou (Inactive)
              rchang Robert Chang
              Votes:
              12 Vote for this issue
              Watchers:
              32 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: