Uploaded image for project: 'Bamboo'
  1. Bamboo
  2. BAM-21157

bouncycastle throw NPE after upgrade to java 8 u272. linux agent fails to connect or UI won't load with ERR_SSL_PROTOCOL_ERROR

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • Highest
    • Resolution: Fixed
    • 6.10.4, 7.0.4
    • 7.2.0
    • Agents
    • None

    Description

      Issue Summary

      After upgrading Java to JDK 8u272 remote and elastic agents fail to connect to Bamboo server. There is often a NullPointerException during the SSL handshake. If connecting to Bamboo via a Tomcat SSL listener, the UI may fail to load with ERR_SSL_PROTOCOL_ERROR (in Chrome).

      This appears to be NPE in PSSSignatureSpi#init due to unchecked null SecureRandom · Issue #633 · bcgit/bc-java · GitHub and fixed with the bouncy castle 1.65. Bamboo currently ships with Bouncy castle 1.64

      Steps to Reproduce

      1. Using Java 8 u262
      2. Configure a Tomcat HTTPS listener
      3. Configure Bamboo remote agents to use SSL: Securing your remote agents - Bamboo Server 7.1 - Atlassian Documentation (this step is required so that the ActiveMQ broker loads BouncyCastle as a provider)
        • This occurs with Bamboo's automatic JMS SSL management, not only custom certs.
      4. Everything is working at this point
      5. Upgrade Java on Bamboo server to JDK8 u272 and restart Bamboo

      Expected Results

      Agent successfully exchanges ssl keys and connects.
      HTTPS connections are successful to Tomcat HTTPS listener.

      Actual Results

      Agents will eventually (after they give up retrying in ~10mins of hanging at Registering with server) fail with the below in $BAMBOO_AGENT_HOME/atlassian-bamboo-agent.log:

      INFO   | jvm 1    | 2020/11/17 14:32:39 | 2020-11-17 03:32:39,895 TRACE [ActiveMQ Transport: ssl://local/192.168.212.131:54663] [FailoverTransport] unconnected handleTransportFailure: javax.net.ssl.SSLException: Received fatal alert: internal_error
      INFO   | jvm 1    | 2020/11/17 14:32:39 | javax.net.ssl.SSLException: Received fatal alert: internal_error
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.Alert.createSSLException(Alert.java:133)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.Alert.createSSLException(Alert.java:117)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.TransportContext.fatal(TransportContext.java:311)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at java.io.DataInputStream.readInt(DataInputStream.java:387)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
      INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at java.lang.Thread.run(Thread.java:748)
      

      Server-side exceptions similar to the below will show in $BAMBOO_HOME/logs/atlassian-bamboo.log

      2020-11-11 08:31:57,330 ERROR [ActiveMQ BrokerService[bamboo] Task-14] [TransportConnector] Could not accept connection from tcp://192.168.1.131:41716 : javax.net.ssl.SSLException
      

      The below can be observed server-side in $BAMBOO_HOME/logs/atlassian-bamboo.log when connecting to the Tomcat SSL connector or with -Djavax.net.debug=all enabled to see the issue with ActiveMQ SSL (agents JMS):

      java.lang.NullPointerException
        	at org.bouncycastle.crypto.signers.PSSSigner.generateSignature(Unknown Source)
        	at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineSign(Unknown Source)
        	at java.security.Signature$Delegate.engineSign(Signature.java:1382)
        	at java.security.Signature.sign(Signature.java:698)
        	at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:181)
        	at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeProducer.produce(ECDHServerKeyExchange.java:499)
        	at sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1020)
        	at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:727)
        	at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:693)
        	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377)
        	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
        	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
        	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)
        	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149)
        	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143)
        	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054)
        	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394)
        	at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708)
        	at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72)
        	at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:961)
        	at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
        	at java.io.DataOutputStream.flush(DataOutputStream.java:123)
        	at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194)
        	at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335)
        	at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317)
        	at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:181)
        	at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:84)
        	at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:74)
        	at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64)
        	at org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1072)
        	at org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:218)
        	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        	at java.lang.Thread.run(Thread.java:748)}
      

      Elastic agents may show:

      2021-03-03 20:40:19,743 ERROR [tunnelserver:26224-1-thread-6] [TunnelAcceptor] Error while accepting tunnel connections.
      javax.net.ssl.SSLException: Received fatal alert: internal_error
      at sun.security.ssl.Alert.createSSLException(Alert.java:133)
      at sun.security.ssl.Alert.createSSLException(Alert.java:117)
      at sun.security.ssl.TransportContext.fatal(TransportContext.java:311)
      at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
      at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185)
      at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149)
      at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143)
      at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054)
      at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394)
      at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708)
      at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72)
      at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791)
      at java.io.InputStream.read(InputStream.java:101)
      at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:62)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
      at java.lang.Thread.run(Thread.java:748)
      

      Workaround

      NOTE: The following workarounds are applicable on Bamboo Server and Bamboo Elastic Agents.

      Option 1 - Rollback Java

      Rollback Java JDK8u272 to a previous version of JDK8. Also required for Bamboo Elastic Agents.

      Option 2 - Disable RSASSA-PSS algorithm

      Bamboo Server:

      • Edit the $JAVA_HOME/jre/lib/security/java.security and add RSASSA-PSS to the jdk.tls.disabledAlgorithms:
        jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \
        EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS
        
      • Or you can optionally create another file with the jdk.tls.disabledAlgorithms values listed above and refer to it by adding the following property to the $BAMBOO_INSTALL/bin/setenv.sh, in JVM_SUPPORT_RECOMMENDED_ARGS
        -Djava.security.properties=/path/to/custom/java.security
        

      Bamboo Elastic Agent:

      • Please read https://bugs.openjdk.java.net/browse/JDK-8226374 to understand why we need to change the Elastic Agent java settings
      • On the Elastic Agent, edit the $JAVA_HOME/jre/lib/security/java.security and add RSASSA-PSS to the jdk.tls.disabledAlgorithms
        jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \
        EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS
      • It is also possible to use a custom java.security file as listed on the Bamboo Server example above. As each Elastic Agent setup is unique we'd rather leave this to the customer's discretion.

       

      This alone will fix the connection between agents and the server via the JMS broker – Bamboo Server only.

      If you are experiencing SSL error with HTTPS requests directly to Tomcat, in addition to disabling RSASSA-PSS above, you'll also need to disable TLSv1.3 by setting sslEnabledProtocols="TLSv1.2". Example:

      <Connector
                  port="8443"
                  maxThreads="150" minSpareThreads="25"
                  connectionTimeout="20000"
                  disableUploadTimeout="true"
                  acceptCount="100"
                  enableLookups="false"
                  maxHttpHeaderSize="8192"
                  useBodyEncodingForURI="true"
                  URIEncoding="UTF-8"
                  keystoreFile="/path/to/keystore.p12"
                  keystorePass="changeit"
                  scheme="https" secure="true" SSLEnabled="true" sslProtocol="TLS" 
                  
                  sslEnabledProtocols="TLSv1.2"
      
                  clientAuth="false" />
      

      Attachments

        Issue Links

          Activity

            People

              8e64d0eeb173 Jan Kolanowski (Inactive)
              cberry@atlassian.com Chris Berry
              Votes:
              2 Vote for this issue
              Watchers:
              12 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: