Loading...

XML

Word

Printable

Type: Bug
Resolution: Unresolved
Priority: Low
Fix Version/s: None
Affects Version/s: 5.8.18, 5.10.4, 6.6.4
Component/s: Data Center - Core
Labels:
- affects-server
- pse-request

Support reference count:
16
Symptom Severity:
Severity 3 - Minor
UIS:
4

Summary

Confluence Application Status Servlet return HTTP 200 when the node is in cluster panic mode and the header Accept-Language is missing. Hazelcast at this point is not active for this node, and having an http status check on a load balancer will cause this check to fail and still consider the node to be active while it is not.

Environment

Confluence Data Center with Load Balancer (this was tested with HAProxy)

Steps to Reproduce

Configure Confluence Data Center with more than 1 node

Configure HAProxy with

option httpchk GET /confluence/status
http-check expect status 200

Try to trigger a panic mode for one of the nodes

Expected Results

Two options:

The ApplicationStatusServlet will return an HTTP 500 error and HAProxy will stop redirecting requests to the node in Panic
The ApplicationStatusServlet will return an HTTP 200 OK with the status page reflecting the status STOPPED of the node

Actual Results

HAProxy is still redirecting requests to that node because it received an HTTP 200. On the other hand, accessing the status page from the browser does indeed return an empty page with a 500 HTTP response code

This is an example of a call initiated from CURL:

curl -i -X GET <base-url>/status

Response:

HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
X-ASEN: SEN-L123456
X-Confluence-Cluster-Node: d1111f1
Content-Type: application/json;charset=ISO-8859-1
Content-Length: 19
Date: Fri, 04 Nov 2016 11:12:02 GMT

On the other hand, simulating the call from the browser, the following header is specifically added to the request

curl -i -X GET -H 'Accept-Language: en-US,en;q=0.8' <base-url>/status

Response:

HTTP/1.1 500 Internal Server Error
Server: Apache-Coyote/1.1
Set-Cookie: JSESSIONID=319B13A07E775E53BBAD5E72445B2308; Path=/; HttpOnly
Content-Type: text/html
Content-Length: 0
Date: Fri, 04 Nov 2016 11:10:14 GMT
Connection: close

The following stacktrace is shown in the logs:

2016-10-26 08:08:08,140 INFO [http-nio-8090-exec-50] [atlassian.confluence.status.SystemErrorInformationLogger] writeToLog 
Request Unique ID : e1bb128b-8d2d-4785-86ee-ffe7e76803c9
--------------------------
JVM Stats
--------------------------
usedMemory = 12341234
usedMemoryInMegabytes = 1234
availableHeap = 1234123
freeMemoryInMegabytes = 1234
allocatedHeap = 12341234
freeAllocatedHeap = 12341234
totalMemory = 12341234
totalMemoryInMegabytes = 1234
availablePermGen = 0
maxPermGen = -1
maxHeap = 12341234
usedHeap = 12341234
freeMemory = 12341234
usedPermGen = -1
--------------------------
Request Information
--------------------------
URL: <confluence-base-url>/confluence/500page.jsp
Scheme: http
Server: <confluence-server>
Port: 8090
URI: /confluence/500page.jsp
Context Path: /confluence
Servlet Path: /500page.jsp
Path Info: null
Query String: null
--------------------------
Attributes
--------------------------
javax.servlet.forward.request_uri: /confluence/status
javax.servlet.forward.context_path: /confluence
javax.servlet.forward.servlet_path: /status
javax.servlet.error.status_code: 500
atlassian.core.seraph.original.url: /500page.jsp
com.opensymphony.sitemesh.APPLIED_ONCE: true
javax.servlet.error.message: 
javax.servlet.error.servlet_name: status-servlet
javax.servlet.error.request_uri: /confluence/status
loginfilter.already.filtered: true
com.atlassian.core.filters.HeaderSanitisingFilter_already_filtered: true
com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFixupFilter: true
javax.servlet.error.exception: com.atlassian.cache.CacheException: Problem retrieving a value from cache com.atlassian.confluence.locale.requestLang
os_securityfilter_already_filtered: true
com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFilter: true
--------------------------
Parameters
--------------------------
caused by: com.atlassian.cache.CacheException: Problem retrieving a value from cache com.atlassian.confluence.locale.requestLang
at com.atlassian.cache.hazelcast.HazelcastCache.getOrLoad(HazelcastCache.java:255)
caused by: com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
at com.hazelcast.spi.AbstractDistributedObject.getService(AbstractDistributedObject.java:93)

2016-10-26 08:08:08,140 ERROR [http-nio-8090-exec-50] [[Standalone].[localhost].[/confluence].[jsp]] log Servlet.service() for servlet jsp threw exception
com.hazelcast.core.HazelcastInstanceNotActiveException: Hazelcast instance is not active!
	at com.hazelcast.spi.AbstractDistributedObject.getService(AbstractDistributedObject.java:93)
	at com.hazelcast.map.impl.proxy.MapProxySupport.toData(MapProxySupport.java:1046)
	at com.hazelcast.map.impl.proxy.MapProxyImpl.get(MapProxyImpl.java:83)
	at com.atlassian.cache.hazelcast.HazelcastCache.getOrLoad(HazelcastCache.java:231)
	at com.atlassian.cache.hazelcast.HazelcastCache.get(HazelcastCache.java:90)
	at com.atlassian.confluence.cache.DefaultConfluenceCache.get(DefaultConfluenceCache.java:52)
	at com.atlassian.confluence.cache.ConfluenceMonitoringCache.get(ConfluenceMonitoringCache.java:49)
	at com.atlassian.confluence.setup.bandana.ConfluenceCachingBandanaPersister.retrieve(ConfluenceCachingBandanaPersister.java:57)
	at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32)
	at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:24)
	at com.atlassian.confluence.setup.settings.DefaultSettingsManager.getGlobalSettings(DefaultSettingsManager.java:43)
	at com.atlassian.confluence.status.service.DefaultSystemInformationService.getConfluenceInfo(DefaultSystemInformationService.java:90)
	at org.apache.jsp._500page_jsp._jspService(_500page_jsp.java:309)
	at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
	at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:438)
	at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:396)
	at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:340)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:729)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:291)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:50)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
	at com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFixupFilter.doFilter(PrettyUrlsSiteMeshFixupFilter.java:36)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
	at com.atlassian.prettyurls.filter.PrettyUrlsDispatcherFilter.doFilter(PrettyUrlsDispatcherFilter.java:60)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
	at com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFilter.doFilter(PrettyUrlsSiteMeshFilter.java:92)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
	at com.atlassian.prettyurls.filter.PrettyUrlsMatcherFilter.doFilter(PrettyUrlsMatcherFilter.java:56)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
	at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
	at com.atlassian.confluence.util.profiling.ProfilingSiteMeshFilter.doFilter(ProfilingSiteMeshFilter.java:46)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
	at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:240)
	at com.atlassian.confluence.web.filter.ConfluenceSecurityFilter.doFilter(ConfluenceSecurityFilter.java:27)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:148)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
	at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58)
	at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70)
	at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:50)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:239)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
	at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:720)
	at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:468)
	at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:391)
	at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:318)
	at org.apache.catalina.core.StandardHostValve.custom(StandardHostValve.java:439)
	at org.apache.catalina.core.StandardHostValve.status(StandardHostValve.java:305)
	at org.apache.catalina.core.StandardHostValve.throwable(StandardHostValve.java:399)
	at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:180)
	at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)
	at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:88)
	at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:518)
	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:673)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1500)
	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1456)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Thread.java:745)

Workaround

The available workaround is to modify the haproxy configuration to include the Accept Language header in the request

option httpchk GET /status HTTP/1.0\r\nAccept-Language:en-GB,en;q=0.8,en-US;q=0.6
http-check expect status 200

HTTP 1.1 is not compatible in this case and it only works with HTTP 1.0

is related to

CONFSERVER-45193 ApplicationStatusServlet creates a session for a failed request

Closed

mentioned in: Page Loading...

Assignee:: Unassigned
Reporter:: Rudy Slaiby
Votes:: 11 Vote for this issue
Watchers:: 17 Start watching this issue

Created:: 26/Oct/2016 3:04 PM
Updated:: 01/Oct/2025 2:08 AM

Details

Description

Summary

Environment

Steps to Reproduce

Expected Results

Actual Results

Workaround

Attachments

Issue Links

Activity

People

Dates