Uploaded image for project: 'Crowd Data Center'
  1. Crowd Data Center
  2. CWD-4968

Tomcat7 produces corrupted gzip response with newest zlib (1.2.11) library

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Low Low
    • 3.0.2, 3.1.1
    • 2.10.1, 3.0.1
    • None
    • None

      After upgrading to the newest zlib library (seen with 1.2.11) , it seems that only first chunk of Tomcat gzip response is compressed, which results, for example, with CSS not being loaded by browser:

      As the response served by Tomcat is invalid, most browser will also terminate the connection resulting in an error like:

      Caused by: org.apache.catalina.connector.ClientAbortException: java.net.SocketException: Broken pipe (Write failed)
      	at org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:407)
      	at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
      	at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:310)
      	at org.apache.catalina.connector.OutputBuffer.writeByte(OutputBuffer.java:451)
      	at org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:77)
      	at com.atlassian.plugin.webresource.impl.annotators.SemicolonResourceContentAnnotator.afterResourceInBatch(SemicolonResourceContentAnnotator.java:53)
      	at com.atlassian.plugin.webresource.impl.annotators.ListOfAnnotators.afterResourceInBatch(ListOfAnnotators.java:35)
      	at com.atlassian.plugin.webresource.impl.helpers.ResourceServingHelpers$1.writeTo(ResourceServingHelpers.java:130)
      	... 102 more
      Caused by: java.net.SocketException: Broken pipe (Write failed)
      	at java.net.SocketOutputStream.socketWrite0(Native Method)
      	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:111)
      	at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
      	at org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
      	at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
      	at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:366)
      	at org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:240)
      	at org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:117)
      	at org.apache.coyote.http11.filters.GzipOutputFilter$FakeOutputStream.write(GzipOutputFilter.java:177)
      	at org.apache.coyote.http11.filters.FlushableGZIPOutputStream.deflate(FlushableGZIPOutputStream.java:153)
      

      on the server side.

      Workaround:

      • if possible downgrade zlib to a version before 1.2.11 (1.2.8 is known to work correctly)
      • alternately disabling compression in Tomcat's server.xml configuration file will alleviate this issue (by setting the attribute compression to off on the Connector element in CROWD_INSTALL_DIR/apache-tomcat/conf/server.xml):
        <?xml version="1.0" encoding="UTF-8"?>
        <Server port="8020" shutdown="SHUTDOWN">
        
            <Service name="Catalina">
        
                <Connector
        ...
                           compression="off"
        ...
        

            [CWD-4968] Tomcat7 produces corrupted gzip response with newest zlib (1.2.11) library

            I think it was the Jessie version, which was for some reason held back in my recent stretch upgrade– unfortunately, it looks like it may be the same  version. 

            apt-cache shows this as the Jessie version:

                zlib1g | 1:1.2.8.dfsg-2+b1 | http://ftp2.de.debian.org/debian jessie/main amd64 Packages

            Sorry to not have the precise info – but here's the data of what I did:

            • upgraded from jessie to stretch, upgraded crowd from 3.0.1 to 3.1.1 shortly thereafter
            • crowd exhibited gzip failure issue (appearing an non-styled pages)
            • turned compression off in server.xml (issue fixed)
            • noticed non-matching zlib version (I believe it's the jessie version listed above, but not 100% sure)– upgraded
            • changed server.xml file to turn compression back on
            • restarted crowd
            • correct behavior observed

            I hope that helps.

             

            Deleted Account (Inactive) added a comment - I think it was the Jessie version, which was for some reason held back in my recent stretch upgrade– unfortunately, it looks like it may be the same  version.  apt-cache shows this as the Jessie version:     zlib1g | 1:1.2.8.dfsg-2+b1 | http://ftp2.de.debian.org/debian jessie/main amd64 Packages Sorry to not have the precise info – but here's the data of what I did: upgraded from jessie to stretch, upgraded crowd from 3.0.1 to 3.1.1 shortly thereafter crowd exhibited gzip failure issue (appearing an non-styled pages) turned compression off in server.xml (issue fixed) noticed non-matching zlib version (I believe it's the jessie version listed above, but not 100% sure)– upgraded changed server.xml file to turn compression back on restarted crowd correct behavior observed I hope that helps.  

            Thanks ryandb for your update. Would you mind sharing the version you were running previously? It would be valuable for us to do additional testing. It might also help other customers running into this issue.

             

            Best regards,

            Marcin Kempa 

            Marcin Kempa added a comment - Thanks ryandb for your update. Would you mind sharing the version you were running previously? It would be valuable for us to do additional testing. It might also help other customers running into this issue.   Best regards, Marcin Kempa 

            looks like zlib might have been the issue — upgraded to  1:1.2.8.dfsg-5 amd64  and behavior is now as expected with compression on.

            Thanks for your help!

                    

             

             

            Deleted Account (Inactive) added a comment - looks like zlib might have been the issue — upgraded to  1:1.2.8.dfsg-5 amd64  and behavior is now as expected with compression on. Thanks for your help!             

            Patryk added a comment -

            Hello ryandb,

            I was unable to reproduce the issue with Crowd 3.1.1 (tar.gz distribution) on Debian 9.2 I've tested with:

            • Oracle JDK8u152
            • Oracle JDK8u151
            • Oracle JDK8u112
            • OpenJDK 8u151

            My zlib1g package version is 1:1.2.8.dfsg-5.

            In order to help us reproduce your issue I'll ask some more questions:

            • What JDK version are you using?
            • What version of zlib do you have installed?
            • Are you also facing this issue when connecting to Crowd directly (without Apache on the way)?
            • Could you provide your Crowd installation directory? Your installation directory will not contain sensitive data unless you have JNDI resources configured or your Crowd home is nested inside your Crowd installation directory
            • Could you provide a HAR file for a request where you're encountering the issue?

            Best regards,
            Patryk Petrowski

            Patryk added a comment - Hello ryandb , I was unable to reproduce the issue with Crowd 3.1.1 (tar.gz distribution) on Debian 9.2 I've tested with: Oracle JDK8u152 Oracle JDK8u151 Oracle JDK8u112 OpenJDK 8u151 My zlib1g package version is 1:1.2.8.dfsg-5. In order to help us reproduce your issue I'll ask some more questions: What JDK version are you using? What version of zlib do you have installed? Are you also facing this issue when connecting to Crowd directly (without Apache on the way)? Could you provide your Crowd installation directory? Your installation directory will not contain sensitive data unless you have JNDI resources configured or your Crowd home is nested inside your Crowd installation directory Could you provide a HAR file for a request where you're encountering the issue? Best regards, Patryk Petrowski

             

            Crowd 3.1.1 - tar.gz distrib  — Apache (SSL ) Proxy setup

            Debian Stretch (9.2) - recently upgraded from Jessie (8.6)


            Known-good (previous) config was Debian Jessie + Crowd 3.0.1.

             

             

            Deleted Account (Inactive) added a comment -   Crowd 3.1.1 - tar.gz distrib  — Apache (SSL ) Proxy setup Debian Stretch (9.2) - recently upgraded from Jessie (8.6) Known-good (previous) config was Debian Jessie + Crowd 3.0.1.    

            Patryk added a comment -

            Hello ryandb,

            Could you please share more details about your setup? Which OS, which type of Crowd distribution and which Tomcat version (if you're using the Crowd WAR distribution)?

            Best regards,
            Patryk Petrowski

            Patryk added a comment - Hello ryandb , Could you please share more details about your setup? Which OS, which type of Crowd distribution and which Tomcat version (if you're using the Crowd WAR distribution)? Best regards, Patryk Petrowski

            This appears to still be an issue in Crowd 3.1.1

            Deleted Account (Inactive) added a comment - This appears to still be an issue in Crowd 3.1.1

            Patryk added a comment -

            Hello,

            Tomcat compression is disabled by default as a workaround for this issue in Crowd 3.0.2. A proper fix is coming in Crowd 3.1.0.

            Best regards,
            Patryk Petrowski

            Patryk added a comment - Hello, Tomcat compression is disabled by default as a workaround for this issue in Crowd 3.0.2. A proper fix is coming in Crowd 3.1.0. Best regards, Patryk Petrowski

              Unassigned Unassigned
              lpater Lukasz Pater
              Affected customers:
              0 This affects my team
              Watchers:
              8 Start watching this issue

                Created:
                Updated:
                Resolved: