-
Bug
-
Resolution: Fixed
-
Highest
-
5.10.8, 6.1.4, 6.6.4, 6.7.0, 6.9.1, 6.11.0, 6.13.6, 6.15.2
-
95
-
Severity 3 - Minor
-
206
-
Issue Summary
Logs flooded with "Broken pipe" errors
Known environment where issue is observed:
- When using firefox 59.0.2+ (64-bit)
- Issue not observed on Google Chrome and on and older Mozilla Firefox ESR 52.7.2 (64bit)
- Using NGINX in front on Confluence as reverse proxy.
- Did not observe issue when using HAProxy load balancer
- Using Confluence Data Center behind AWS ALB
- Using Confluence 6.10.0 behind reverse proxy (WAF - Alert Logic) handling ssl offload.
Steps to Reproduce
- Have at least one of the above environment mentioned met
- Start Confluence
- Access Confluence starting page
- Login to application and view/edit any pages
Expected Results
Logs are not flooded.
Actual Results
The below exception is thrown in the atlassian-confluence.log file during login and page viewing/editing
2018-03-29 13:11:04,014 ERROR [http-nio-8090-exec-1] [[Standalone].[localhost].[/].[noop]] log Servlet.service() for servlet [noop] in context with path [] threw exception java.lang.RuntimeException: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe at com.atlassian.plugin.cache.filecache.impl.StreamsCache.streamFromFile(StreamsCache.java:140) at com.atlassian.plugin.cache.filecache.impl.OneStreamCache.stream(OneStreamCache.java:65) at com.atlassian.plugin.cache.filecache.impl.FileCacheImpl.cache(FileCacheImpl.java:100) at com.atlassian.plugin.webresource.impl.http.Controller.sendCachedInProduction(Controller.java:355) at com.atlassian.plugin.webresource.impl.http.Controller.sendCached(Controller.java:305) at com.atlassian.plugin.webresource.impl.http.Controller.serveResources(Controller.java:234) at com.atlassian.plugin.webresource.impl.http.Controller.serveBatch(Controller.java:111) at com.atlassian.plugin.webresource.impl.http.Router$5.apply(Router.java:92) at com.atlassian.plugin.webresource.impl.http.Router$5.apply(Router.java:83) at com.atlassian.plugin.webresource.impl.support.http.BaseRouter.callHandler(BaseRouter.java:197) at com.atlassian.plugin.webresource.impl.support.http.BaseRouter.dispatch(BaseRouter.java:163) at com.atlassian.plugin.webresource.servlet.PluginResourceDownload.serveFile(PluginResourceDownload.java:65) at com.atlassian.plugin.servlet.AbstractFileServerServlet.doGet(AbstractFileServerServlet.java:28) at javax.servlet.http.HttpServlet.service(HttpServlet.java:622) at javax.servlet.http.HttpServlet.service(HttpServlet.java:729) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:292) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.confluence.plugins.baseurl.IncludeResourcesFilter.doFilter(IncludeResourcesFilter.java:52) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.labs.botkiller.BotKillerFilter.doFilter(BotKillerFilter.java:36) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:24) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.confluence.efi.OnboardingFilter.doFilter(OnboardingFilter.java:61) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFixupFilter.doFilter(PrettyUrlsSiteMeshFixupFilter.java:36) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.prettyurls.filter.PrettyUrlsDispatcherFilter.doFilter(PrettyUrlsDispatcherFilter.java:60) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.prettyurls.filter.PrettyUrlsSiteMeshFilter.doFilter(PrettyUrlsSiteMeshFilter.java:92) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.prettyurls.filter.PrettyUrlsMatcherFilter.doFilter(PrettyUrlsMatcherFilter.java:56) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.core.filters.ServletContextThreadLocalFilter.doFilter(ServletContextThreadLocalFilter.java:21) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.confluence.util.UserNameHeaderFilter.doFilter(UserNameHeaderFilter.java:25) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:242) at com.atlassian.confluence.web.filter.ConfluenceSecurityFilter.doFilter(ConfluenceSecurityFilter.java:25) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:148) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:67) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.prettyurls.filter.PrettyUrlsCombinedMatchDispatcherFilter.doFilter(PrettyUrlsCombinedMatchDispatcherFilter.java:61) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter.doFilter(DelegatingPluginFilter.java:64) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:37) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:70) at com.atlassian.plugin.servlet.filter.ServletFilterModuleContainerFilter.doFilter(ServletFilterModuleContainerFilter.java:58) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.confluence.web.ConfluenceJohnsonFilter.doFilter(ConfluenceJohnsonFilter.java:30) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.confluence.web.filter.DebugFilter.doFilter(DebugFilter.java:46) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:716) at org.apache.catalina.core.ApplicationDispatcher.processRequest(ApplicationDispatcher.java:466) at org.apache.catalina.core.ApplicationDispatcher.doForward(ApplicationDispatcher.java:391) at org.apache.catalina.core.ApplicationDispatcher.forward(ApplicationDispatcher.java:318) at com.atlassian.confluence.servlet.rewrite.CachedRewriteMatch.execute(CachedRewriteMatch.java:32) at org.tuckey.web.filters.urlrewrite.RewrittenUrlClass.doRewrite(RewrittenUrlClass.java:61) at org.tuckey.web.filters.urlrewrite.RuleChain.handleRewrite(RuleChain.java:171) at org.tuckey.web.filters.urlrewrite.RuleChain.doRules(RuleChain.java:145) at org.tuckey.web.filters.urlrewrite.UrlRewriter.processRequest(UrlRewriter.java:92) at org.tuckey.web.filters.urlrewrite.UrlRewriteFilter.doFilter(UrlRewriteFilter.java:389) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.confluence.cache.TransactionalCacheFactoryCleanupFilter.doFilter(TransactionalCacheFactoryCleanupFilter.java:22) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.core.filters.ServletContextThreadLocalFilter.doFilter(ServletContextThreadLocalFilter.java:21) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.confluence.util.UserLoggingContextFilter.doFilter(UserLoggingContextFilter.java:32) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.confluence.util.UserNameHeaderFilter.doFilter(UserNameHeaderFilter.java:25) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.confluence.web.filter.MauEventFilter.doFilter(MauEventFilter.java:42) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.confluence.util.UserThreadLocalFilter.doFilter(UserThreadLocalFilter.java:39) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.confluence.web.filter.ConfluenceTimeoutFilter.doFilter(ConfluenceTimeoutFilter.java:57) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:242) at com.atlassian.confluence.web.filter.ConfluenceSecurityFilter.doFilter(ConfluenceSecurityFilter.java:25) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.confluence.web.filter.ThreadLocalCacheFilter.doFilter(ThreadLocalCacheFilter.java:25) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:103) at com.atlassian.confluence.util.AbstractBootstrapHotSwappingFilter.doFilter(AbstractBootstrapHotSwappingFilter.java:35) at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:148) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:240) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:207) at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:39) at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:58) at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:67)
or
An established connection was aborted by the software in your host machine
Or
An existing connection was forcibly closed by the remote host
Known Workaround
Cosider one of the following options.
Option 1
If using NGINX as a reverse proxy in front of Confluence, try to disable buffering by adding the following directive associated to Confluence:
proxy_buffering off;
Option 2
Clean the logs with the following command:
|sed -r '/ERROR/{N;s/(.*)\n(.*ERROR.*)/\1\n\n\2/}'|awk '/ERROR/{ getline nextline; if (nextline ~ /^[^2C].*Exception: java.io.IOException: Broken pipe/) d = 1; else print $0 "\n" nextline; next} d && !/^([[:blank:]]|Caused by)/ {d = 0} !d'
Option 3
If you are sure these messages are not related to any problem to Confluence user, you may use log4j configuration to filter them out as described in Configuring log4j to filter out unwanted messages in Confluence application log.
- is duplicated by
-
CONFSERVER-42905 Broken pipe exceptions in Confluence 5.10
- Closed
-
CONFSERVER-45362 Broken pipe exceptions in Confluence 5.10.8 and above
- Closed
- links to
- mentioned in
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...