Loading...

Details

Type: Bug
Resolution: Fixed
Priority: Highest
Fix Version/s: 7.2.0
Affects Version/s: 6.10.4, 7.0.4
Component/s: Agents
Labels:
None

Support reference count:
8
Symptom Severity:
Severity 1 - Critical
UIS:
3
Bug Fix Policy:
View Atlassian Server bug fix policy

Description

Issue Summary

After upgrading Java to JDK 8u272 remote and elastic agents fail to connect to Bamboo server. There is often a NullPointerException during the SSL handshake. If connecting to Bamboo via a Tomcat SSL listener, the UI may fail to load with ERR_SSL_PROTOCOL_ERROR (in Chrome).

This appears to be NPE in PSSSignatureSpi#init due to unchecked null SecureRandom · Issue #633 · bcgit/bc-java · GitHub and fixed with the bouncy castle 1.65. Bamboo currently ships with Bouncy castle 1.64

Steps to Reproduce

Using Java 8 u262
Configure a Tomcat HTTPS listener
Configure Bamboo remote agents to use SSL: Securing your remote agents - Bamboo Server 7.1 - Atlassian Documentation (this step is required so that the ActiveMQ broker loads BouncyCastle as a provider)
- This occurs with Bamboo's automatic JMS SSL management, not only custom certs.
Everything is working at this point
Upgrade Java on Bamboo server to JDK8 u272 and restart Bamboo

Expected Results

Agent successfully exchanges ssl keys and connects.
HTTPS connections are successful to Tomcat HTTPS listener.

Actual Results

Agents will eventually (after they give up retrying in ~10mins of hanging at Registering with server) fail with the below in $BAMBOO_AGENT_HOME/atlassian-bamboo-agent.log:

INFO   | jvm 1    | 2020/11/17 14:32:39 | 2020-11-17 03:32:39,895 TRACE [ActiveMQ Transport: ssl://local/192.168.212.131:54663] [FailoverTransport] unconnected handleTransportFailure: javax.net.ssl.SSLException: Received fatal alert: internal_error
INFO   | jvm 1    | 2020/11/17 14:32:39 | javax.net.ssl.SSLException: Received fatal alert: internal_error
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.Alert.createSSLException(Alert.java:133)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.Alert.createSSLException(Alert.java:117)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.TransportContext.fatal(TransportContext.java:311)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at java.io.DataInputStream.readInt(DataInputStream.java:387)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215)
INFO   | jvm 1    | 2020/11/17 14:32:39 | 	at java.lang.Thread.run(Thread.java:748)

Server-side exceptions similar to the below will show in $BAMBOO_HOME/logs/atlassian-bamboo.log

2020-11-11 08:31:57,330 ERROR [ActiveMQ BrokerService[bamboo] Task-14] [TransportConnector] Could not accept connection from tcp://192.168.1.131:41716 : javax.net.ssl.SSLException

The below can be observed server-side in $BAMBOO_HOME/logs/atlassian-bamboo.log when connecting to the Tomcat SSL connector or with -Djavax.net.debug=all enabled to see the issue with ActiveMQ SSL (agents JMS):

java.lang.NullPointerException
  	at org.bouncycastle.crypto.signers.PSSSigner.generateSignature(Unknown Source)
  	at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineSign(Unknown Source)
  	at java.security.Signature$Delegate.engineSign(Signature.java:1382)
  	at java.security.Signature.sign(Signature.java:698)
  	at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:181)
  	at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeProducer.produce(ECDHServerKeyExchange.java:499)
  	at sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1020)
  	at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:727)
  	at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:693)
  	at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377)
  	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
  	at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
  	at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)
  	at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149)
  	at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143)
  	at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054)
  	at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394)
  	at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708)
  	at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72)
  	at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:961)
  	at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115)
  	at java.io.DataOutputStream.flush(DataOutputStream.java:123)
  	at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194)
  	at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335)
  	at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317)
  	at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:181)
  	at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:84)
  	at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:74)
  	at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64)
  	at org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1072)
  	at org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:218)
  	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
  	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
  	at java.lang.Thread.run(Thread.java:748)}

Elastic agents may show:

2021-03-03 20:40:19,743 ERROR [tunnelserver:26224-1-thread-6] [TunnelAcceptor] Error while accepting tunnel connections.
javax.net.ssl.SSLException: Received fatal alert: internal_error
at sun.security.ssl.Alert.createSSLException(Alert.java:133)
at sun.security.ssl.Alert.createSSLException(Alert.java:117)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:311)
at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185)
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149)
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143)
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394)
at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708)
at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72)
at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791)
at java.io.InputStream.read(InputStream.java:101)
at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:62)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

Workaround

NOTE: The following workarounds are applicable on Bamboo Server and Bamboo Elastic Agents.

Option 1 - Rollback Java

Rollback Java JDK8u272 to a previous version of JDK8. Also required for Bamboo Elastic Agents.

Option 2 - Disable RSASSA-PSS algorithm

Bamboo Server:

Edit the $JAVA_HOME/jre/lib/security/java.security and add RSASSA-PSS to the jdk.tls.disabledAlgorithms:

jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \
EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS

Or you can optionally create another file with the jdk.tls.disabledAlgorithms values listed above and refer to it by adding the following property to the $BAMBOO_INSTALL/bin/setenv.sh, in JVM_SUPPORT_RECOMMENDED_ARGS
```
-Djava.security.properties=/path/to/custom/java.security
```

Bamboo Elastic Agent:

Please read https://bugs.openjdk.java.net/browse/JDK-8226374 to understand why we need to change the Elastic Agent java settings

On the Elastic Agent, edit the $JAVA_HOME/jre/lib/security/java.security and add RSASSA-PSS to the jdk.tls.disabledAlgorithms

jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \
EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS

It is also possible to use a custom java.security file as listed on the Bamboo Server example above. As each Elastic Agent setup is unique we'd rather leave this to the customer's discretion.

This alone will fix the connection between agents and the server via the JMS broker – Bamboo Server only.

If you are experiencing SSL error with HTTPS requests directly to Tomcat, in addition to disabling RSASSA-PSS above, you'll also need to disable TLSv1.3 by setting sslEnabledProtocols="TLSv1.2". Example:

<Connector
            port="8443"
            maxThreads="150" minSpareThreads="25"
            connectionTimeout="20000"
            disableUploadTimeout="true"
            acceptCount="100"
            enableLookups="false"
            maxHttpHeaderSize="8192"
            useBodyEncodingForURI="true"
            URIEncoding="UTF-8"
            keystoreFile="/path/to/keystore.p12"
            keystorePass="changeit"
            scheme="https" secure="true" SSLEnabled="true" sslProtocol="TLS" 
            
            sslEnabledProtocols="TLSv1.2"

            clientAuth="false" />

Attachments

Issue Links

Mentioned in

Bamboo KTLO: BAM-21157

mentioned in: Page Loading...

bouncycastle throw NPE after upgrade to java 8 u272. linux agent fails to connect or UI won't load with ERR_SSL_PROTOCOL_ERROR