Details
-
Bug
-
Resolution: Unresolved
-
Low
-
None
-
5.8.18, 5.10.4, 6.6.4
-
4
-
Severity 3 - Minor
-
1
-
Description
Summary
Confluence Application Status Servlet return HTTP 200 when the node is in cluster panic mode and the header Accept-Language is missing. Hazelcast at this point is not active for this node, and having an http status check on a load balancer will cause this check to fail and still consider the node to be active while it is not.
Environment
Confluence Data Center with Load Balancer (this was tested with HAProxy)
Steps to Reproduce
- Configure Confluence Data Center with more than 1 node
- Configure HAProxy with
option httpchk GET /confluence/status http-check expect status 200
- Try to trigger a panic mode for one of the nodes
Expected Results
Two options:
- The ApplicationStatusServlet will return an HTTP 500 error and HAProxy will stop redirecting requests to the node in Panic
- The ApplicationStatusServlet will return an HTTP 200 OK with the status page reflecting the status STOPPED of the node
Actual Results
HAProxy is still redirecting requests to that node because it received an HTTP 200. On the other hand, accessing the status page from the browser does indeed return an empty page with a 500 HTTP response code
This is an example of a call initiated from CURL:
curl -i -X GET <base-url>/status
Response:
HTTP/1.1 200 OK Server: Apache-Coyote/1.1 X-ASEN: SEN-L123456 X-Confluence-Cluster-Node: d1111f1 Content-Type: application/json;charset=ISO-8859-1 Content-Length: 19 Date: Fri, 04 Nov 2016 11:12:02 GMT
On the other hand, simulating the call from the browser, the following header is specifically added to the request
curl -i -X GET -H 'Accept-Language: en-US,en;q=0.8' <base-url>/status
Response:
HTTP/1.1 500 Internal Server Error Server: Apache-Coyote/1.1 Set-Cookie: JSESSIONID=319B13A07E775E53BBAD5E72445B2308; Path=/; HttpOnly Content-Type: text/html Content-Length: 0 Date: Fri, 04 Nov 2016 11:10:14 GMT Connection: close
The following stacktrace is shown in the logs:
2016-10-26 08:08:08,140 INFO [http-nio-8090-exec-50] [atlassian.confluence.status.SystemErrorInformationLogger] writeToLog Request Unique ID : e1bb128b-8d2d-4785-86ee-ffe7e76803c9 -------------------------- JVM Stats -------------------------- usedMemory = 12341234 usedMemoryInMegabytes = 1234 availableHeap = 1234123 freeMemoryInMegabytes = 1234 allocatedHeap = 12341234 freeAllocatedHeap = 12341234 totalMemory = 12341234 totalMemoryInMegabytes = 1234 availablePermGen = 0 maxPermGen = -1 maxHeap = 12341234 usedHeap = 12341234 freeMemory = 12341234 usedPermGen = -1 -------------------------- Request Information -------------------------- URL: <confluence-base-url>/confluence/500page.jsp Scheme: http Server: <confluence-server> Port: 8090 URI: /confluence/500page.jsp Context Path: /confluence Servlet Path: /500page.jsp Path Info: null Query String: null -------------------------- Attributes -------------------------- javax.servlet.forward.request_uri: /confluence/status javax.servlet.forward.context_path: /confluence javax.servlet.forward.servlet_path: /status javax.servlet.error.status_code: 500 atlassian.core.seraph.original.url: /500page.jsp com.opensymphony.sitemesh.APPLIED_ONCE: true javax.servlet.error.message: javax.servlet.error.servlet_name: status-servlet javax.servlet.error.request_uri: /confluence/status loginfilter.already.filtered: true com.atlassian.core.filters.HeaderSanitisingFilter_already_filtered: true com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFixupFilter: true javax.servlet.error.exception: com.atlassian.cache.CacheException: Problem retrieving a value from cache com.atlassian.confluence.locale.requestLang os_securityfilter_already_filtered: true com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFilter: true -------------------------- Parameters -------------------------- caused by: com.atlassian.cache.CacheException: Problem retrieving a value from cache com.atlassian.confluence.locale.requestLang at com.atlassian.cache.hazelcast.HazelcastCache.getOrLoad(HazelcastCache.java:255) caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active! at com.hazelcast.spi.AbstractDistributedObject.getService(AbstractDistributedObject.java:93) 2016-10-26 08:08:08,140 ERROR [http-nio-8090-exec-50] [[Standalone].[localhost].[/confluence].[jsp]] log Servlet.service() for servlet jsp threw exception com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active! at com.hazelcast.spi.AbstractDistributedObject.getService(AbstractDistributedObject.java:93) at com.hazelcast.map.impl.proxy.MapProxySupport.toData(MapProxySupport.java:1046) at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:83) at com.atlassian.cache.hazelcast.HazelcastCache.getOrLoad(HazelcastCache.java:231) at com.atlassian.cache.hazelcast.HazelcastCache.get(HazelcastCache.java:90) at com.atlassian.confluence.cache.DefaultConfluenceCache.get(DefaultConfluenceCache.java:52) at com.atlassian.confluence.cache.ConfluenceMonitoringCache.get(ConfluenceMonitoringCache.java:49) at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:57) at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32) at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:24) at com.atlassian.confluence.setup.settings.DefaultSettingsManager.getGlobalSettings(DefaultSettingsManager.java:43) at com.atlassian.confluence.status.service.DefaultSystemInformationService.getConfluenceInfo(DefaultSystemInformationService.java:90) at org.apache.jsp._500page_jsp._jspService(_500page_jsp.java:309) at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438) at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396) at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:50) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFixupFilter.doFilter(PrettyUrlsSiteMeshFixupFilter.java:36) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.prettyurls.filter.PrettyUrlsDispatcherFilter.doFilter(PrettyUrlsDispatcherFilter.java:60) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFilter.doFilter(PrettyUrlsSiteMeshFilter.java:92) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.prettyurls.filter.PrettyUrlsMatcherFilter.doFilter(PrettyUrlsMatcherFilter.java:56) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129) at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77) at com.atlassian.confluence.util.profiling.ProfilingSiteMeshFilter.doFilter(ProfilingSiteMeshFilter.java:46) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:240) at com.atlassian.confluence.web.filter.ConfluenceSecurityFilter.doFilter(ConfluenceSecurityFilter.java:27) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:148) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:50) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:720) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:468) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:391) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:318) at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:439) at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:305) at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:399) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518) at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091) at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:673) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.lang.Thread.run(Thread.java:745)
Workaround
The available workaround is to modify the haproxy configuration to include the Accept Language header in the request
option httpchk GET /status HTTP/1.0\r\nAccept-Language:en-GB,en;q=0.8,en-US;q=0.6 http-check expect status 200
HTTP 1.1 is not compatible in this case and it only works with HTTP 1.0
Attachments
Issue Links
- is related to
-
CONFSERVER-45193 ApplicationStatusServlet creates a session for a failed request
- Closed
- mentioned in
-
Page Loading...