Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-63503

Frequent java.io.IOException: broken pipe messages appearing in logs

XMLWordPrintable

    • 6.04
    • 102
    • Severity 3 - Minor
    • 1,092
    • Hide
      Atlassian Update – 21-12-2018

      The Atlassian team has performed testing that validates that the "Broken pipe" logging output is not the artefact of a bug but rather a generic exception thrown by the application server when a connection is closed unexpectedly (usually on the client side). Our findings show that this exception is a symptom rather than a cause. We have found that one of the most common ways for this issue to appear is when a client closes an open connection while performing any action (like loading a page, or downloading an attachment), this also happens when the connection is slow and in our tests the Broken Pipe error happened even more by forcefully closing certain connections when using a proxy/load balancer like Nginx (like closing the web browser or even simply cancelling a download). This can happen as well with anything that is interrupting a connection between the client and server including performance issues or even having network intermittency. 

      We have updated the KB article ClientAbortException: java.io.IOException: Broken pipe and will continue adding different scenarios and possible workarounds as they get identified.

      Finally, understanding how disruptive this behavior can be, we’ve been working on a solution to address the most common cases where this exception is thrown, and the progress for it can be found at: JRASERVER-68615. As for this Bug request, we will proceed to close it for now. In the case that you experience this issue and it is of concern, please reach out to our support team for them to evaluate your case and determine the next steps.

      Best regards,
      Rene Chiquete - JAC Champion

      Show
      Atlassian Update – 21-12-2018 The Atlassian team has performed testing that validates that the "Broken pipe" logging output is not the artefact of a bug but rather a generic exception thrown by the application server when a connection is closed unexpectedly (usually on the client side). Our findings show that this exception is a symptom rather than a cause. We have found that one of the most common ways for this issue to appear is when a client closes an open connection while performing any action (like loading a page, or downloading an attachment), this also happens when the connection is slow and in our tests the Broken Pipe error happened even more by forcefully closing certain connections when using a proxy/load balancer like Nginx (like closing the web browser or even simply cancelling a download). This can happen as well with anything that is interrupting a connection between the client and server including performance issues or even having network intermittency.  We have updated the KB article ClientAbortException: java.io.IOException: Broken pipe and will continue adding different scenarios and possible workarounds as they get identified. Finally, understanding how disruptive this behavior can be, we’ve been working on a solution to address the most common cases where this exception is thrown, and the progress for it can be found at: JRASERVER-68615 . As for this Bug request, we will proceed to close it for now. In the case that you experience this issue and it is of concern, please reach out to our support team for them to evaluate your case and determine the next steps. Best regards, Rene Chiquete - JAC Champion

      Issue Summary

      After upgrading from JIRA 6.x to JIRA 7.1.x, the following trace can be found in the logs mentioning "Broken Pipe".

      Steps to Reproduce

      1. A connection is established by between the Jira application server and a client (anyone connecting to Jira).
      2. The connection is closed unexpectedly (cancelling download, killing browser, closing tab while page is loading, etc).

      Expected Results

      No problem is perceived and no exception is thrown in logs.

      Actual Results

      The UI does not perceive any problem but the below exception is thrown in the atlasian-jira.log file:

       http-nio-8080-exec-5 ERROR      [o.a.c.c.C.[.[localhost].[/].[default]] Servlet.service() for servlet [default] in context with path [] threw exception
java.lang.RuntimeException: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe

      Notes

      • Sometimes this is thrown so frequent in the logs that it is spamming the logs
      • The most common causes for this exception to appear are covered in our KB article ClientAbortException: java.io.IOException: Broken pipe
      • There are variations on the stack trace with the same exception, like this: 
        ERROR [o.a.c.c.C.[.[localhost].[/].[default]] Servlet.service() for servlet [default] in context with path [] threw exception
java.lang.RuntimeException: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
at com.atlassian.servicedesk.internal.util.scala.ScalaJavaInterOp$1.apply(ScalaJavaInterOp.java:27)
at com.atlassian.servicedesk.internal.utils.context.CustomerContextUtil$.outOfCustomerContext(CustomerContextUtil.scala:48)
at com.atlassian.servicedesk.internal.utils.context.CustomerContextUtil.outOfCustomerContext(CustomerContextUtil.scala)
at com.atlassian.servicedesk.internal.utils.context.CustomerContextServiceImpl.outOfCustomerContext(CustomerContextServiceImpl.java:24)
at com.atlassian.servicedesk.internal.web.CustomerContextSettingFilter.outOfCustomerContext(CustomerContextSettingFilter.java:164)
at com.atlassian.servicedesk.internal.web.CustomerContextSettingFilter.doFilterImpl(CustomerContextSettingFilter.java:120)
at com.atlassian.servicedesk.internal.web.CustomerContextSettingFilter.doFilter(CustomerContextSettingFilter.java:112)
... 55 filtered
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.catalina.connector.ClientAbortException: java.io.IOException: Broken pipe
at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:393)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:426)
at org.apache.catalina.connector.OutputBuffer.doFlush(OutputBuffer.java:342)
at org.apache.catalina.connector.OutputBuffer.flush(OutputBuffer.java:317)
at org.apache.catalina.connector.CoyoteOutputStream.flush(CoyoteOutputStream.java:110)
at com.sun.jersey.spi.container.ContainerResponse$CommittingOutputStream.flush(ContainerResponse.java:146)
at org.codehaus.jackson.impl.Utf8Generator.flush(Utf8Generator.java:1085)
at org.codehaus.jackson.map.ObjectMapper.writeValue(ObjectMapper.java:1606)
at org.codehaus.jackson.jaxrs.JacksonJsonProvider.writeTo(JacksonJsonProvider.java:558)
at com.sun.jersey.spi.container.ContainerResponse.write(ContainerResponse.java:302)
... 5 filtered
at com.atlassian.plugins.rest.module.RestDelegatingServletFilter$JerseyOsgiServletContainer.doFilter(RestDelegatingServletFilter.java:154)
... 1 filtered
at com.atlassian.plugins.rest.module.RestDelegatingServletFilter.doFilter(RestDelegatingServletFilter.java:68)
... 91 filtered
at com.atlassian.jira.security.JiraSecurityFilter.doFilter(JiraSecurityFilter.java:70)
... 16 filtered
at com.atlassian.plugins.rest.module.servlet.RestSeraphFilter.doFilter(RestSeraphFilter.java:37)
... 22 filtered
at com.atlassian.servicedesk.internal.web.CustomerContextSettingFilter.lambda$invokeFilterChain$0(CustomerContextSettingFilter.java:169)
at com.atlassian.servicedesk.internal.util.scala.ScalaJavaInterOp$1.apply(ScalaJavaInterOp.java:25)
... 65 more
Caused by: java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:471)

      Workaround

      Currently there is no known workaround for this behavior. A workaround will be added here when available

        1. broken-pipe-100660631-primary.idge.jpg
          broken-pipe-100660631-primary.idge.jpg
          110 kB

              Unassigned Unassigned
              mmcmahon Matthew McMahon (Inactive)
              Votes:
              397 Vote for this issue
              Watchers:
              334 Start watching this issue

                Created:
                Updated:
                Resolved: