-
Type:
Bug
-
Resolution: Incorrectly Filed
-
Priority:
Medium
-
None
-
Affects Version/s: 7.4.18, 7.13.9, 7.19.1
-
Component/s: Macros - Gadgets
-
None
-
Severity 2 - Major
Issue Summary
If Confluence has been integrated with a Jira(or some other instance over an application link) and also contains gadgets on pages, Confluence performance can degrade if Jira(or connected instance) has an outage or performance issue. This is due to how Confluence currently stores and fetches gadget data from Jira. This issue is similar to CONFSERVER-57696 and is specific to items added under "Gadgets Feed" and not "Gadgets Specification"
This is reproducible on Data Center: Yes
Steps to Reproduce
- Integrate Confluence & Jira
- Set up some Jira Gadgets in Confluence under Gadget Feeds
- Make Jira unresponsive to Confluence
- Visit a page with a Jira gadget on it
Expected Results
Confluence may not render the gadget (because Jira is unresponsive) but performance across the instance will remain stable
Actual Results
Confluence threads can get stuck in a RUNNABLE state attempting to read gadget data from Jira. This has the unfortunate affect of bottlenecking other HTTP threads as those threads take forever to process and can cause a performance issue. With time, more and more keeps moving to this state and might exhaust all the http threads configured under Tomcat and post this, we are unable to access Confluence.
Stuck threads can be seen under the catalina logs with below stack trace -
at java.base@11.0.16/jdk.internal.misc.Unsafe.park(Native Method)
at java.base@11.0.16/java.util.concurrent.locks.LockSupport.park(LockSupport.java:194)
at java.base@11.0.16/java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2081)
at org.codehaus.httpcache4j.cache.Mutex.acquire(Mutex.java:45)
at org.codehaus.httpcache4j.cache.HTTPCache.execute(HTTPCache.java:87)
at org.codehaus.httpcache4j.cache.HTTPCache.execute(HTTPCache.java:66)
at com.atlassian.gadgets.directory.internal.GadgetHttpCache.execute(GadgetHttpCache.java:38)
at com.atlassian.gadgets.directory.internal.impl.GadgetFeedReaderImpl.<init>(GadgetFeedReaderImpl.java:45)
at com.atlassian.gadgets.directory.internal.impl.GadgetFeedReaderFactoryImpl.getFeedReader(GadgetFeedReaderFactoryImpl.java:26)
at com.atlassian.gadgets.directory.internal.GadgetFeedsSpecProvider.getFeedReaders(GadgetFeedsSpecProvider.java:76)
at com.atlassian.gadgets.directory.internal.GadgetFeedsSpecProvider.contains(GadgetFeedsSpecProvider.java:51)
at jdk.internal.reflect.GeneratedMethodAccessor980.invoke(Unknown Source)
at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
...
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy2411.contains(Unknown Source)
at com.atlassian.gadgets.directory.internal.impl.GadgetSpecProviderDirectoryEntryProvider.lambda$contains$0(GadgetSpecProviderDirectoryEntryProvider.java:62)
at com.atlassian.gadgets.directory.internal.impl.GadgetSpecProviderDirectoryEntryProvider$$Lambda$7840/0x0000000805663440.test(Unknown Source)
at java.base@11.0.16/java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
at java.base@11.0.16/java.util.Spliterators$IteratorSpliterator.tryAdvance(Spliterators.java:1812)
at java.base@11.0.16/java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:127)
at java.base@11.0.16/java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:502)
at java.base@11.0.16/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:488)
at java.base@11.0.16/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
at java.base@11.0.16/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
at java.base@11.0.16/java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
at java.base@11.0.16/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.base@11.0.16/java.util.stream.ReferencePipeline.anyMatch(ReferencePipeline.java:528)
at com.atlassian.gadgets.directory.internal.impl.GadgetSpecProviderDirectoryEntryProvider.contains(GadgetSpecProviderDirectoryEntryProvider.java:60)
at com.atlassian.gadgets.directory.internal.impl.DirectoryImpl$2.apply(DirectoryImpl.java:65)
at com.atlassian.gadgets.directory.internal.impl.DirectoryImpl$2.apply(DirectoryImpl.java:62)
at com.google.common.collect.Iterators.indexOf(Iterators.java:764)
at com.google.common.collect.Iterators.any(Iterators.java:663)
at com.google.common.collect.Iterables.any(Iterables.java:608)
at com.atlassian.gadgets.directory.internal.impl.DirectoryImpl.contains(DirectoryImpl.java:40)
at com.atlassian.gadgets.directory.internal.impl.GadgetSpecUrlRenderPermissionImpl.voteOn(GadgetSpecUrlRenderPermissionImpl.java:34)
at jdk.internal.reflect.GeneratedMethodAccessor6710.invoke(Unknown Source)
at java.base@11.0.16/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base@11.0.16/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
at com.sun.proxy.$Proxy2417.voteOn(Unknown Source)
at com.atlassian.gadgets.renderer.internal.GadgetSpecUrlCheckerImpl.assertRenderable(GadgetSpecUrlCheckerImpl.java:60)
at com.atlassian.gadgets.renderer.internal.servlet.GadgetSpecUrlRenderPermissionServletFilter.doFilter(GadgetSpecUrlRenderPermissionServletFilter.java:87)
at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:62)
....
Workaround
Remove specific Gadgets Feed(if feasible) from the instance that are unavailable or experiencing performance issues and then restart Confluence instances. Also, if this is happening with Jira instance or any other Atlassian product instance, you can add the product gadget feed as a part of "Gadget specification" and remove it from here. Post this, Confluence restart would still be needed to remove the existing threads around these gadget feeds.