Details
-
Bug
-
Resolution: Fixed
-
Medium
-
1
-
Severity 2 - Major
-
Description
It was observed in that during JIRA performance degradation, Confluence can gradually become unresponsive due to threads apparently attempting to read gadget feeds from JIRA.
Thread dumps taken from the Confluence JVM during this time will show the majority of HTTP threads (245 out of 300 in the case observed) looking like the below:
"http-nio-8090-exec-232" #424 daemon prio=5 os_prio=0 tid=0x00007f4aac0d0000 nid=0xca62 waiting on condition [0x00007f47c49c4000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000003cd4014d0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) at org.codehaus.httpcache4j.cache.Mutex.acquire(Mutex.java:45) at org.codehaus.httpcache4j.cache.HTTPCache.doCachedRequest(HTTPCache.java:103) at org.codehaus.httpcache4j.cache.HTTPCache.doCachedRequest(HTTPCache.java:80) at com.atlassian.gadgets.directory.internal.impl.GadgetFeedReaderImpl.<init>(GadgetFeedReaderImpl.java:46) at com.atlassian.gadgets.directory.internal.impl.GadgetFeedReaderFactoryImpl.getFeedReader(GadgetFeedReaderFactoryImpl.java:28) at sun.reflect.GeneratedMethodAccessor3143.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:307) at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:58) at org.springframework.osgi.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:62) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:56) at org.springframework.osgi.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:39) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) at org.springframework.osgi.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:59) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:131) at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:119) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy2822.getFeedReader(Unknown Source) at com.atlassian.confluence.plugins.gadgets.metadata.GadgetsMacroMetadataProvider.getUrisFromFeedStore(GadgetsMacroMetadataProvider.java:240) at com.atlassian.confluence.plugins.gadgets.metadata.GadgetsMacroMetadataProvider.getGadgetUris(GadgetsMacroMetadataProvider.java:225) at com.atlassian.confluence.plugins.gadgets.metadata.CachingGadgetsMacroMetadataProvider.getGadgetUris(CachingGadgetsMacroMetadataProvider.java:181) at com.atlassian.confluence.plugins.gadgets.metadata.CachingGadgetsMacroMetadataProvider.getData(CachingGadgetsMacroMetadataProvider.java:116) at com.atlassian.confluence.plugins.gadgets.metadata.CachingGadgetsMacroMetadataProvider.getSummaries(CachingGadgetsMacroMetadataProvider.java:157) at com.atlassian.confluence.macro.browser.DefaultMacroMetadataManager.buildMacroSummaries(DefaultMacroMetadataManager.java:123) at com.atlassian.confluence.macro.browser.DefaultMacroMetadataManager.getAllMacroSummaries(DefaultMacroMetadataManager.java:42) at com.atlassian.confluence.macro.browser.DefaultMacroBrowserManager.getMacroSummaries(DefaultMacroBrowserManager.java:66) at com.atlassian.confluence.macro.browser.actions.BrowseMacrosAction.getSummaries(BrowseMacrosAction.java:58) at com.atlassian.confluence.macro.browser.actions.BrowseMacrosAction.execute(BrowseMacrosAction.java:36) at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:168) at com.opensymphony.xwork.interceptor.AroundInterceptor.intercept(AroundInterceptor.java:35) at com.opensymphony.xwork.DefaultActionInvocation.invoke(DefaultActionInvocation.java:165) at com.atlassian.confluence.core.ConfluenceWorkflowInterceptor.intercept(ConfluenceWorkflowInterceptor.java:65) ... at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501) at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:950) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1040) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1721) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1679) - locked <0x000000048d4da578> (a org.apache.tomcat.util.net.NioChannel) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)
These threads will be long-running and appear in multiple thread dumps taken over a period of time. The impact to users is that eventually Confluence will not have enough threads free in order to service users.
Expected Behaviour
There should be a limit for number of requests for applink communication to avoid performance on other application. Especially if the other application is not responsive.
Workaround
Effort should be made to address the JIRA performance problem as that is the trigger for this bug. In the short term, the JIRA gadget feed may be temporarily removed from Confluence Admin > External Gadgets in order to avoid further outages.
Attachments
Issue Links
- is cloned from
-
-
- Closed
-
- is related to
-
-
- Closed
-