Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-57696

Confluence can degrade if Jira is unresponsive due to gadget feed retrieval requests

    XMLWordPrintable

Details

    Description

      Summary

      If Confluence has been integrated with a Jira instance and also contains gadgets on pages, Confluence performance can degrade if Jira has an outage or performance issue. This is due to how Confluence currently stores and fetches gadget data from Jira.

      Environment

      • Confluence
      • Jira
      • Confluence Gadgets

      Steps to Reproduce

      1. Integrate Confluence & Jira
      2. Set up some Jira Gadgets in Confluence
      3. Make Jira unresponsive to Confluence
      4. Visit a page with a Jira gadget on it

      Expected Results

      1. Confluence may not render the gadget (because Jira is unresponsive) but performance across the instance will remain stable

      Actual Results

      1. Confluence threads can get stuck in a RUNNABLE state attempting to read gadget data from Jira. This has the unfortunate affect of bottlenecking other HTTP threads as those threads take forever to process and can cause a performance issue.

      Workaround

      Unknown

      Diagnosis:

      You may see just a few long running RUNNABLE threads with other threads stacking up behind these threads causing a bottleneck.

       171 threaddump-node6/sync_threads.1544641605.txt-   java.lang.Thread.State: WAITING (parking)
        14 threaddump-node6/sync_threads.1544641605.txt-   java.lang.Thread.State: RUNNABLE
      

      From RUNNABLE threads, you may see some like the following:

      "http-nio-8090-exec-4766" #210557 daemon prio=5 os_prio=0 tid=0x00007fefbc25e800 nid=0xf080 runnable [0x00007fee4066e000]
         java.lang.Thread.State: RUNNABLE
      	at java.net.SocketInputStream.socketRead0(Native Method)
      	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
      ...
      	at org.apache.shindig.gadgets.DefaultGadgetSpecFactory.getGadgetSpec(DefaultGadgetSpecFactory.java:91)
      	at com.atlassian.gadgets.renderer.internal.local.LocalGadgetSpecFactory.getGadgetSpec(LocalGadgetSpecFactory.java:80)
      	at com.atlassian.gadgets.renderer.internal.local.LocalGadgetSpecFactory.getGadgetSpec(LocalGadgetSpecFactory.java:59)
      	at com.atlassian.gadgets.renderer.internal.GadgetSpecFactoryImpl.getGadgetSpec(GadgetSpecFactoryImpl.java:128)
      	at com.atlassian.gadgets.renderer.internal.GadgetSpecFactoryImpl.getGadgetSpec(GadgetSpecFactoryImpl.java:79)
      ...
      	at com.atlassian.confluence.plugins.gadgets.GadgetMacro.getViewType(GadgetMacro.java:460)
      	at com.atlassian.confluence.plugins.gadgets.GadgetMacro.getGadgetView(GadgetMacro.java:436)
      	at com.atlassian.confluence.plugins.gadgets.GadgetMacro.generateMacroDiv(GadgetMacro.java:190)
      	at com.atlassian.confluence.plugins.gadgets.GadgetMacro.renderGadget(GadgetMacro.java:172)
      	at com.atlassian.confluence.plugins.gadgets.GadgetMacro.execute(GadgetMacro.java:157)
      	at com.atlassian.confluence.content.render.xhtml.view.macro.ViewMacroMarshaller.executeMacro(ViewMacroMarshaller.java:273)
      	at com.atlassian.confluence.pages.actions.ViewPageAction.execute(ViewPageAction.java:270)
      

      Or Stuck Threads:

      [ContainerBackgroundProcessor[StandardEngine[Catalina]]] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread "ajp-nio-127.0.0.1-8009-exec-412 url:/plugins/macrob...owse-macros.action" (id=xxx) has been active for 60,733 milliseconds (since 4/22/20 9:29 AM) to serve the same request for https://test/plugins/macrobrowser/browse-macros.action?detailed=false&macroMetadataClientCacheKey=1587562106899 and may be stuck (configured threshold for this StuckThreadDetectionValve is 60 seconds). There is/are 252 thread(s) in total that are monitored by this Valve and may be stuck.
       java.lang.Throwable
      	at sun.misc.Unsafe.park(Native Method)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
      	at org.codehaus.httpcache4j.cache.Mutex.acquire(Mutex.java:45)
      	at org.codehaus.httpcache4j.cache.HTTPCache.doCachedRequest(HTTPCache.java:103)
      	at org.codehaus.httpcache4j.cache.HTTPCache.doCachedRequest(HTTPCache.java:80)
      	at com.atlassian.gadgets.directory.internal.impl.GadgetFeedReaderImpl.<init>(GadgetFeedReaderImpl.java:43)
      	at com.atlassian.gadgets.directory.internal.impl.GadgetFeedReaderFactoryImpl.getFeedReader(GadgetFeedReaderFactoryImpl.java:28)
      	at sun.reflect.GeneratedMethodAccessor949.invoke(Unknown Source)
      	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      	at java.lang.reflect.Method.invoke(Method.java:498)
      	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
      	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
      	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
      	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
      	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
      	at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
      	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
      	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
      	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
      	at com.sun.proxy.$Proxy2053.getFeedReader(Unknown Source)
      	at com.atlassian.confluence.plugins.gadgets.metadata.GadgetsMacroMetadataProvider.getUrisFromFeedStore(GadgetsMacroMetadataProvider.java:243) 

      Fetching gadget spec doesn't currently have a timeout threshold which explains why these threads can get stuck in a running state.

      Attachments

        Issue Links

          Activity

            People

              glipatov George Lipatov
              nlopez@atlassian.com Nancy Lopez
              Votes:
              13 Vote for this issue
              Watchers:
              28 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: