Highest Issue Summary
After upgrading Java to JDK 8u272 remote and elastic agents fail to connect to Bamboo server. There is often a NullPointerException during the SSL handshake. If connecting to Bamboo via a Tomcat SSL listener, the UI may fail to load with ERR_SSL_PROTOCOL_ERROR (in Chrome).
This appears to be NPE in PSSSignatureSpi#init due to unchecked null SecureRandom · Issue #633 · bcgit/bc-java · GitHub and fixed with the bouncy castle 1.65. Bamboo currently ships with Bouncy castle 1.64
Steps to Reproduce
- Using Java 8 u262
- Configure a Tomcat HTTPS listener
- Configure Bamboo remote agents to use SSL: Securing your remote agents - Bamboo Server 7.1 - Atlassian Documentation (this step is required so that the ActiveMQ broker loads BouncyCastle as a provider)
- This occurs with Bamboo's automatic JMS SSL management, not only custom certs.
- Everything is working at this point
- Upgrade Java on Bamboo server to JDK8 u272 and restart Bamboo
Expected Results
Agent successfully exchanges ssl keys and connects.
HTTPS connections are successful to Tomcat HTTPS listener.
Actual Results
Agents will eventually (after they give up retrying in ~10mins of hanging at Registering with server) fail with the below in $BAMBOO_AGENT_HOME/atlassian-bamboo-agent.log:
INFO | jvm 1 | 2020/11/17 14:32:39 | 2020-11-17 03:32:39,895 TRACE [ActiveMQ Transport: ssl://local/192.168.212.131:54663] [FailoverTransport] unconnected handleTransportFailure: javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:133) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:117) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.io.DataInputStream.readInt(DataInputStream.java:387) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.lang.Thread.run(Thread.java:748)
Server-side exceptions similar to the below will show in $BAMBOO_HOME/logs/atlassian-bamboo.log
2020-11-11 08:31:57,330 ERROR [ActiveMQ BrokerService[bamboo] Task-14] [TransportConnector] Could not accept connection from tcp://192.168.1.131:41716 : javax.net.ssl.SSLException
The below can be observed server-side in $BAMBOO_HOME/logs/atlassian-bamboo.log when connecting to the Tomcat SSL connector or with -Djavax.net.debug=all enabled to see the issue with ActiveMQ SSL (agents JMS):
java.lang.NullPointerException at org.bouncycastle.crypto.signers.PSSSigner.generateSignature(Unknown Source) at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineSign(Unknown Source) at java.security.Signature$Delegate.engineSign(Signature.java:1382) at java.security.Signature.sign(Signature.java:698) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:181) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeProducer.produce(ECDHServerKeyExchange.java:499) at sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1020) at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:727) at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:693) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:961) at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194) at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335) at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:181) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:84) at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:74) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64) at org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1072) at org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:218) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)}
Elastic agents may show:
2021-03-03 20:40:19,743 ERROR [tunnelserver:26224-1-thread-6] [TunnelAcceptor] Error while accepting tunnel connections. javax.net.ssl.SSLException: Received fatal alert: internal_error at sun.security.ssl.Alert.createSSLException(Alert.java:133) at sun.security.ssl.Alert.createSSLException(Alert.java:117) at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) at java.io.InputStream.read(InputStream.java:101) at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:62) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)
Workaround
NOTE: The following workarounds are applicable on Bamboo Server and Bamboo Elastic Agents.
Option 1 - Rollback Java
Rollback Java JDK8u272 to a previous version of JDK8. Also required for Bamboo Elastic Agents.
Option 2 - Disable RSASSA-PSS algorithm
Bamboo Server:
- Edit the $JAVA_HOME/jre/lib/security/java.security and add RSASSA-PSS to the jdk.tls.disabledAlgorithms:
jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS
- Or you can optionally create another file with the jdk.tls.disabledAlgorithms values listed above and refer to it by adding the following property to the $BAMBOO_INSTALL/bin/setenv.sh, in JVM_SUPPORT_RECOMMENDED_ARGS
-Djava.security.properties=/path/to/custom/java.security
Bamboo Elastic Agent:
- Please read https://bugs.openjdk.java.net/browse/JDK-8226374 to understand why we need to change the Elastic Agent java settings
- On the Elastic Agent, edit the $JAVA_HOME/jre/lib/security/java.security and add RSASSA-PSS to the jdk.tls.disabledAlgorithms
jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS
- It is also possible to use a custom java.security file as listed on the Bamboo Server example above. As each Elastic Agent setup is unique we'd rather leave this to the customer's discretion.
This alone will fix the connection between agents and the server via the JMS broker – Bamboo Server only.
If you are experiencing SSL error with HTTPS requests directly to Tomcat, in addition to disabling RSASSA-PSS above, you'll also need to disable TLSv1.3 by setting sslEnabledProtocols="TLSv1.2". Example:
<Connector
port="8443"
maxThreads="150" minSpareThreads="25"
connectionTimeout="20000"
disableUploadTimeout="true"
acceptCount="100"
enableLookups="false"
maxHttpHeaderSize="8192"
useBodyEncodingForURI="true"
URIEncoding="UTF-8"
keystoreFile="/path/to/keystore.p12"
keystorePass="changeit"
scheme="https" secure="true" SSLEnabled="true" sslProtocol="TLS"
sslEnabledProtocols="TLSv1.2"
clientAuth="false" />
- Mentioned in
- mentioned in
-
Page Failed to load
| Form Name | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
[BAM-21157] bouncycastle throw NPE after upgrade to java 8 u272. linux agent fails to connect or UI won't load with ERR_SSL_PROTOCOL_ERROR
| Remote Link | Original: This issue links to "Page (Confluence)" [ 966258 ] |
| Remote Link | New: This issue links to "Page (Confluence)" [ 966258 ] |
| Remote Link | New: This issue links to "Page (Confluence)" [ 716273 ] |
| Description |
Original:
h3. Issue Summary
After upgrading Java to JDK 8u272 remote and elastic agents fail to connect to Bamboo server. There is often a NullPointerException during the SSL handshake. If connecting to Bamboo via a Tomcat SSL listener, the UI may fail to load with {{ERR_SSL_PROTOCOL_ERROR}} (in Chrome). This appears to be [NPE in PSSSignatureSpi#init due to unchecked null SecureRandom · Issue #633 · bcgit/bc-java · GitHub |https://github.com/bcgit/bc-java/issues/633] and fixed with the bouncy castle 1.65. Bamboo currently ships with Bouncy castle 1.64 h3. Steps to Reproduce # Using Java 8 u262 # Configure a Tomcat HTTPS listener # Configure Bamboo remote agents to use SSL: [Securing your remote agents - Bamboo Server 7.1 - Atlassian Documentation|https://confluence.atlassian.com/bamboo/securing-your-remote-agents-289277197.html] (this step is required so that the ActiveMQ broker loads BouncyCastle as a provider) ** This occurs with Bamboo's automatic JMS SSL management, not only custom certs. # Everything is working at this point # Upgrade Java on Bamboo server to JDK8 u272 and restart Bamboo h3. Expected Results Agent successfully exchanges ssl keys and connects. HTTPS connections are successful to Tomcat HTTPS listener. h3. Actual Results Agents will eventually (after they give up retrying in ~10mins of hanging at Registering with server) fail with the below in {{{}$BAMBOO_AGENT_HOME/atlassian-bamboo-agent.log{}}}: {noformat} INFO | jvm 1 | 2020/11/17 14:32:39 | 2020-11-17 03:32:39,895 TRACE [ActiveMQ Transport: ssl://local/192.168.212.131:54663] [FailoverTransport] unconnected handleTransportFailure: javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:133) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:117) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.io.DataInputStream.readInt(DataInputStream.java:387) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.lang.Thread.run(Thread.java:748) {noformat} Server-side exceptions similar to the below will show in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} {code:java} 2020-11-11 08:31:57,330 ERROR [ActiveMQ BrokerService[bamboo] Task-14] [TransportConnector] Could not accept connection from tcp://192.168.1.131:41716 : javax.net.ssl.SSLException {code} The below can be observed server-side in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} when connecting to the Tomcat SSL connector or with {{-Djavax.net.debug=all}} enabled to see the issue with ActiveMQ SSL (agents JMS): {noformat} java.lang.NullPointerException at org.bouncycastle.crypto.signers.PSSSigner.generateSignature(Unknown Source) at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineSign(Unknown Source) at java.security.Signature$Delegate.engineSign(Signature.java:1382) at java.security.Signature.sign(Signature.java:698) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:181) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeProducer.produce(ECDHServerKeyExchange.java:499) at sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1020) at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:727) at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:693) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:961) at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194) at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335) at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:181) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:84) at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:74) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64) at org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1072) at org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:218) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)} {noformat} Elastic agents may show: {noformat} 2021-03-03 20:40:19,743 ERROR [tunnelserver:26224-1-thread-6] [TunnelAcceptor] Error while accepting tunnel connections. javax.net.ssl.SSLException: Received fatal alert: internal_error at sun.security.ssl.Alert.createSSLException(Alert.java:133) at sun.security.ssl.Alert.createSSLException(Alert.java:117) at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) at java.io.InputStream.read(InputStream.java:101) at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:62) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {noformat} h3. Workaround (!) {*}NOTE{*}: The following workarounds are applicable on Bamboo Server and Bamboo Elastic Agents. h5. *Option 1 - Rollback Java* Rollback Java JDK8u272 to a previous version of JDK8. Also required for Bamboo Elastic Agents. h5. *Option 2 - Disable RSASSA-PSS algorithm* *Bamboo Server:* * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{{}jdk.tls.disabledAlgorithms{}}}: {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS {code} * Or you can optionally create another file with the {{jdk.tls.disabledAlgorithms}} values listed above and refer to it by adding the following property to the {{{}$BAMBOO_INSTALL/bin/setenv.sh{}}}, in {{JVM_SUPPORT_RECOMMENDED_ARGS}} {code:java} -Djava.security.properties=/path/to/custom/java.security {code} *Bamboo Elastic Agent:* * Please read [https://bugs.openjdk.java.net/browse/JDK-8226374] to understand why we need to change the Elastic Agent java settings * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS{code} * It is also possible to use a custom {{java.security}} file as listed on the Bamboo Server example above. As each Elastic Agent setup is unique we'd rather leave this to the customer's discretion. This alone will fix the connection between agents and the server via the JMS broker – Bamboo Server only. If you are experiencing SSL error with HTTPS requests directly to Tomcat, {*}in addition to disabling RSASSA-PSS above{*}, you'll also need to disable TLSv1.3 by setting {{{}sslEnabledProtocols="TLSv1.2"{}}}. Example: {noformat} <Connector port="8443" maxThreads="150" minSpareThreads="25" connectionTimeout="20000" disableUploadTimeout="true" acceptCount="100" enableLookups="false" maxHttpHeaderSize="8192" useBodyEncodingForURI="true" URIEncoding="UTF-8" keystoreFile="/path/to/keystore.p12" keystorePass="changeit" scheme="https" secure="true" SSLEnabled="true" sslProtocol="TLS" sslEnabledProtocols="TLSv1.2" clientAuth="false" /> {noformat} |
New:
h3. Issue Summary
After upgrading Java to JDK 8u272 remote and elastic agents fail to connect to Bamboo server. There is often a NullPointerException during the SSL handshake. If connecting to Bamboo via a Tomcat SSL listener, the UI may fail to load with {{ERR_SSL_PROTOCOL_ERROR}} (in Chrome). This appears to be [NPE in PSSSignatureSpi#init due to unchecked null SecureRandom · Issue #633 · bcgit/bc-java · GitHub |https://github.com/bcgit/bc-java/issues/633] and fixed with the bouncy castle 1.65. Bamboo currently ships with Bouncy castle 1.64 h3. Steps to Reproduce # Using Java 8 u262 # Configure a Tomcat HTTPS listener # Configure Bamboo remote agents to use SSL: [Securing your remote agents - Bamboo Server 7.1 - Atlassian Documentation|https://confluence.atlassian.com/bamboo/securing-your-remote-agents-289277197.html] (this step is required so that the ActiveMQ broker loads BouncyCastle as a provider) ** This occurs with Bamboo's automatic JMS SSL management, not only custom certs. # Everything is working at this point # Upgrade Java on Bamboo server to JDK8 u272 and restart Bamboo h3. Expected Results Agent successfully exchanges ssl keys and connects. HTTPS connections are successful to Tomcat HTTPS listener. h3. Actual Results Agents will eventually (after they give up retrying in ~10mins of hanging at Registering with server) fail with the below in {{{}$BAMBOO_AGENT_HOME/atlassian-bamboo-agent.log{}}}: {noformat} INFO | jvm 1 | 2020/11/17 14:32:39 | 2020-11-17 03:32:39,895 TRACE [ActiveMQ Transport: ssl://local/192.168.212.131:54663] [FailoverTransport] unconnected handleTransportFailure: javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:133) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:117) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.io.DataInputStream.readInt(DataInputStream.java:387) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.lang.Thread.run(Thread.java:748) {noformat} Server-side exceptions similar to the below will show in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} {code:java} 2020-11-11 08:31:57,330 ERROR [ActiveMQ BrokerService[bamboo] Task-14] [TransportConnector] Could not accept connection from tcp://192.168.1.131:41716 : javax.net.ssl.SSLException {code} The below can be observed server-side in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} when connecting to the Tomcat SSL connector or with {{-Djavax.net.debug=all}} enabled to see the issue with ActiveMQ SSL (agents JMS): {noformat} java.lang.NullPointerException at org.bouncycastle.crypto.signers.PSSSigner.generateSignature(Unknown Source) at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineSign(Unknown Source) at java.security.Signature$Delegate.engineSign(Signature.java:1382) at java.security.Signature.sign(Signature.java:698) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:181) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeProducer.produce(ECDHServerKeyExchange.java:499) at sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1020) at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:727) at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:693) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:961) at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194) at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335) at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:181) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:84) at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:74) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64) at org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1072) at org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:218) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)} {noformat} Elastic agents may show: {noformat} 2021-03-03 20:40:19,743 ERROR [tunnelserver:26224-1-thread-6] [TunnelAcceptor] Error while accepting tunnel connections. javax.net.ssl.SSLException: Received fatal alert: internal_error at sun.security.ssl.Alert.createSSLException(Alert.java:133) at sun.security.ssl.Alert.createSSLException(Alert.java:117) at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) at java.io.InputStream.read(InputStream.java:101) at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:62) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {noformat} h3. Workaround (!) {*}NOTE{*}: The following workarounds are applicable on Bamboo Server and Bamboo Elastic Agents. h5. *Option 1 - Rollback Java* Rollback Java JDK8u272 to a previous version of JDK8. Also required for Bamboo Elastic Agents. h5. *Option 2 - Disable RSASSA-PSS algorithm* *Bamboo Server:* * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{{}jdk.tls.disabledAlgorithms{}}}: {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS {code} * Or you can optionally create another file with the {{jdk.tls.disabledAlgorithms}} values listed above and refer to it by adding the following property to the {{{}$BAMBOO_INSTALL/bin/setenv.sh{}}}, in {{JVM_SUPPORT_RECOMMENDED_ARGS}} {code:java} -Djava.security.properties=/path/to/custom/java.security {code} *Bamboo Elastic Agent:* * Please read [https://bugs.openjdk.java.net/browse/JDK-8226374] to understand why we need to change the Elastic Agent java settings * On the Elastic Agent, edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS{code} * It is also possible to use a custom {{java.security}} file as listed on the Bamboo Server example above. As each Elastic Agent setup is unique we'd rather leave this to the customer's discretion. This alone will fix the connection between agents and the server via the JMS broker – Bamboo Server only. If you are experiencing SSL error with HTTPS requests directly to Tomcat, {*}in addition to disabling RSASSA-PSS above{*}, you'll also need to disable TLSv1.3 by setting {{{}sslEnabledProtocols="TLSv1.2"{}}}. Example: {noformat} <Connector port="8443" maxThreads="150" minSpareThreads="25" connectionTimeout="20000" disableUploadTimeout="true" acceptCount="100" enableLookups="false" maxHttpHeaderSize="8192" useBodyEncodingForURI="true" URIEncoding="UTF-8" keystoreFile="/path/to/keystore.p12" keystorePass="changeit" scheme="https" secure="true" SSLEnabled="true" sslProtocol="TLS" sslEnabledProtocols="TLSv1.2" clientAuth="false" /> {noformat} |
| Description |
Original:
h3. Issue Summary
After upgrading Java to JDK 8u272 remote and elastic agents fail to connect to Bamboo server. There is often a NullPointerException during the SSL handshake. If connecting to Bamboo via a Tomcat SSL listener, the UI may fail to load with {{ERR_SSL_PROTOCOL_ERROR}} (in Chrome). This appears to be [NPE in PSSSignatureSpi#init due to unchecked null SecureRandom · Issue #633 · bcgit/bc-java · GitHub |https://github.com/bcgit/bc-java/issues/633] and fixed with the bouncy castle 1.65. Bamboo currently ships with Bouncy castle 1.64 h3. Steps to Reproduce # Using Java 8 u262 # Configure a Tomcat HTTPS listener # Configure Bamboo remote agents to use SSL: [Securing your remote agents - Bamboo Server 7.1 - Atlassian Documentation|https://confluence.atlassian.com/bamboo/securing-your-remote-agents-289277197.html] (this step is required so that the ActiveMQ broker loads BouncyCastle as a provider) ** This occurs with Bamboo's automatic JMS SSL management, not only custom certs. # Everything is working at this point # Upgrade Java on Bamboo server to JDK8 u272 and restart Bamboo h3. Expected Results Agent successfully exchanges ssl keys and connects. HTTPS connections are successful to Tomcat HTTPS listener. h3. Actual Results Agents will eventually (after they give up retrying in ~10mins of hanging at Registering with server) fail with the below in {{{}$BAMBOO_AGENT_HOME/atlassian-bamboo-agent.log{}}}: {noformat} INFO | jvm 1 | 2020/11/17 14:32:39 | 2020-11-17 03:32:39,895 TRACE [ActiveMQ Transport: ssl://local/192.168.212.131:54663] [FailoverTransport] unconnected handleTransportFailure: javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:133) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:117) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.io.DataInputStream.readInt(DataInputStream.java:387) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.lang.Thread.run(Thread.java:748) {noformat} Server-side exceptions similar to the below will show in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} {code:java} 2020-11-11 08:31:57,330 ERROR [ActiveMQ BrokerService[bamboo] Task-14] [TransportConnector] Could not accept connection from tcp://192.168.1.131:41716 : javax.net.ssl.SSLException {code} The below can be observed server-side in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} when connecting to the Tomcat SSL connector or with {{-Djavax.net.debug=all}} enabled to see the issue with ActiveMQ SSL (agents JMS): {noformat} java.lang.NullPointerException at org.bouncycastle.crypto.signers.PSSSigner.generateSignature(Unknown Source) at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineSign(Unknown Source) at java.security.Signature$Delegate.engineSign(Signature.java:1382) at java.security.Signature.sign(Signature.java:698) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:181) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeProducer.produce(ECDHServerKeyExchange.java:499) at sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1020) at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:727) at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:693) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:961) at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194) at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335) at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:181) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:84) at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:74) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64) at org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1072) at org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:218) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)} {noformat} Elastic agents may show: {noformat} 2021-03-03 20:40:19,743 ERROR [tunnelserver:26224-1-thread-6] [TunnelAcceptor] Error while accepting tunnel connections. javax.net.ssl.SSLException: Received fatal alert: internal_error at sun.security.ssl.Alert.createSSLException(Alert.java:133) at sun.security.ssl.Alert.createSSLException(Alert.java:117) at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) at java.io.InputStream.read(InputStream.java:101) at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:62) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {noformat} h3. Workaround h5. *Option 1 - Rollback Java* (!) {*}NOTE{*}: For Bamboo Server only Rollback Java JDK8u272 to a previous version of JDK8 h5. *Option 2 - Disable RSASSA-PSS algorithm* (!) {*}NOTE{*}: For Bamboo Server and Elastic Agents *Bamboo Server:* * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} *or* create another file containing: {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS {code} * Add this to the {{{}$BAMBOO_INSTALL/bin/setenv.sh{}}}, in {{JVM_SUPPORT_RECOMMENDED_ARGS}} {code:java} -Djava.security.properties=/path/to/file/created.above {code} {*}Bamboo Elastic Agent:{*} * Please read [https://bugs.openjdk.java.net/browse/JDK-8226374|https://bugs.openjdk.java.net/browse/JDK-8226374] to understand why we need to change the Elastic Agent java settings * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS{code} This alone will fix the connection between agents and the server via the JMS broker – Bamboo Server only. If you are experiencing SSL error with HTTPS requests directly to Tomcat, {*}in addition to disabling RSASSA-PSS above{*}, you'll also need to disable TLSv1.3 by setting {{{}sslEnabledProtocols="TLSv1.2"{}}}. Example: {noformat} <Connector port="8443" maxThreads="150" minSpareThreads="25" connectionTimeout="20000" disableUploadTimeout="true" acceptCount="100" enableLookups="false" maxHttpHeaderSize="8192" useBodyEncodingForURI="true" URIEncoding="UTF-8" keystoreFile="/path/to/keystore.p12" keystorePass="changeit" scheme="https" secure="true" SSLEnabled="true" sslProtocol="TLS" sslEnabledProtocols="TLSv1.2" clientAuth="false" /> {noformat} |
New:
h3. Issue Summary
After upgrading Java to JDK 8u272 remote and elastic agents fail to connect to Bamboo server. There is often a NullPointerException during the SSL handshake. If connecting to Bamboo via a Tomcat SSL listener, the UI may fail to load with {{ERR_SSL_PROTOCOL_ERROR}} (in Chrome). This appears to be [NPE in PSSSignatureSpi#init due to unchecked null SecureRandom · Issue #633 · bcgit/bc-java · GitHub |https://github.com/bcgit/bc-java/issues/633] and fixed with the bouncy castle 1.65. Bamboo currently ships with Bouncy castle 1.64 h3. Steps to Reproduce # Using Java 8 u262 # Configure a Tomcat HTTPS listener # Configure Bamboo remote agents to use SSL: [Securing your remote agents - Bamboo Server 7.1 - Atlassian Documentation|https://confluence.atlassian.com/bamboo/securing-your-remote-agents-289277197.html] (this step is required so that the ActiveMQ broker loads BouncyCastle as a provider) ** This occurs with Bamboo's automatic JMS SSL management, not only custom certs. # Everything is working at this point # Upgrade Java on Bamboo server to JDK8 u272 and restart Bamboo h3. Expected Results Agent successfully exchanges ssl keys and connects. HTTPS connections are successful to Tomcat HTTPS listener. h3. Actual Results Agents will eventually (after they give up retrying in ~10mins of hanging at Registering with server) fail with the below in {{{}$BAMBOO_AGENT_HOME/atlassian-bamboo-agent.log{}}}: {noformat} INFO | jvm 1 | 2020/11/17 14:32:39 | 2020-11-17 03:32:39,895 TRACE [ActiveMQ Transport: ssl://local/192.168.212.131:54663] [FailoverTransport] unconnected handleTransportFailure: javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:133) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:117) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.io.DataInputStream.readInt(DataInputStream.java:387) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.lang.Thread.run(Thread.java:748) {noformat} Server-side exceptions similar to the below will show in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} {code:java} 2020-11-11 08:31:57,330 ERROR [ActiveMQ BrokerService[bamboo] Task-14] [TransportConnector] Could not accept connection from tcp://192.168.1.131:41716 : javax.net.ssl.SSLException {code} The below can be observed server-side in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} when connecting to the Tomcat SSL connector or with {{-Djavax.net.debug=all}} enabled to see the issue with ActiveMQ SSL (agents JMS): {noformat} java.lang.NullPointerException at org.bouncycastle.crypto.signers.PSSSigner.generateSignature(Unknown Source) at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineSign(Unknown Source) at java.security.Signature$Delegate.engineSign(Signature.java:1382) at java.security.Signature.sign(Signature.java:698) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:181) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeProducer.produce(ECDHServerKeyExchange.java:499) at sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1020) at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:727) at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:693) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:961) at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194) at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335) at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:181) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:84) at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:74) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64) at org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1072) at org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:218) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)} {noformat} Elastic agents may show: {noformat} 2021-03-03 20:40:19,743 ERROR [tunnelserver:26224-1-thread-6] [TunnelAcceptor] Error while accepting tunnel connections. javax.net.ssl.SSLException: Received fatal alert: internal_error at sun.security.ssl.Alert.createSSLException(Alert.java:133) at sun.security.ssl.Alert.createSSLException(Alert.java:117) at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) at java.io.InputStream.read(InputStream.java:101) at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:62) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {noformat} h3. Workaround (!) {*}NOTE{*}: The following workarounds are applicable on Bamboo Server and Bamboo Elastic Agents. h5. *Option 1 - Rollback Java* Rollback Java JDK8u272 to a previous version of JDK8. Also required for Bamboo Elastic Agents. h5. *Option 2 - Disable RSASSA-PSS algorithm* *Bamboo Server:* * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{{}jdk.tls.disabledAlgorithms{}}}: {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS {code} * Or you can optionally create another file with the {{jdk.tls.disabledAlgorithms}} values listed above and refer to it by adding the following property to the {{{}$BAMBOO_INSTALL/bin/setenv.sh{}}}, in {{JVM_SUPPORT_RECOMMENDED_ARGS}} {code:java} -Djava.security.properties=/path/to/custom/java.security {code} *Bamboo Elastic Agent:* * Please read [https://bugs.openjdk.java.net/browse/JDK-8226374] to understand why we need to change the Elastic Agent java settings * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS{code} * It is also possible to use a custom {{java.security}} file as listed on the Bamboo Server example above. As each Elastic Agent setup is unique we'd rather leave this to the customer's discretion. This alone will fix the connection between agents and the server via the JMS broker – Bamboo Server only. If you are experiencing SSL error with HTTPS requests directly to Tomcat, {*}in addition to disabling RSASSA-PSS above{*}, you'll also need to disable TLSv1.3 by setting {{{}sslEnabledProtocols="TLSv1.2"{}}}. Example: {noformat} <Connector port="8443" maxThreads="150" minSpareThreads="25" connectionTimeout="20000" disableUploadTimeout="true" acceptCount="100" enableLookups="false" maxHttpHeaderSize="8192" useBodyEncodingForURI="true" URIEncoding="UTF-8" keystoreFile="/path/to/keystore.p12" keystorePass="changeit" scheme="https" secure="true" SSLEnabled="true" sslProtocol="TLS" sslEnabledProtocols="TLSv1.2" clientAuth="false" /> {noformat} |
| Description |
Original:
h3. Issue Summary
After upgrading Java to JDK 8u272 remote and elastic agents fail to connect to Bamboo server. There is often a NullPointerException during the SSL handshake. If connecting to Bamboo via a Tomcat SSL listener, the UI may fail to load with {{ERR_SSL_PROTOCOL_ERROR}} (in Chrome). This appears to be [NPE in PSSSignatureSpi#init due to unchecked null SecureRandom · Issue #633 · bcgit/bc-java · GitHub |https://github.com/bcgit/bc-java/issues/633] and fixed with the bouncy castle 1.65. Bamboo currently ships with Bouncy castle 1.64 h3. Steps to Reproduce # Using Java 8 u262 # Configure a Tomcat HTTPS listener # Configure Bamboo remote agents to use SSL: [Securing your remote agents - Bamboo Server 7.1 - Atlassian Documentation|https://confluence.atlassian.com/bamboo/securing-your-remote-agents-289277197.html] (this step is required so that the ActiveMQ broker loads BouncyCastle as a provider) ** This occurs with Bamboo's automatic JMS SSL management, not only custom certs. # Everything is working at this point # Upgrade Java on Bamboo server to JDK8 u272 and restart Bamboo h3. Expected Results Agent successfully exchanges ssl keys and connects. HTTPS connections are successful to Tomcat HTTPS listener. h3. Actual Results Agents will eventually (after they give up retrying in ~10mins of hanging at Registering with server) fail with the below in {{{}$BAMBOO_AGENT_HOME/atlassian-bamboo-agent.log{}}}: {noformat} INFO | jvm 1 | 2020/11/17 14:32:39 | 2020-11-17 03:32:39,895 TRACE [ActiveMQ Transport: ssl://local/192.168.212.131:54663] [FailoverTransport] unconnected handleTransportFailure: javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:133) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:117) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.io.DataInputStream.readInt(DataInputStream.java:387) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.lang.Thread.run(Thread.java:748) {noformat} Server-side exceptions similar to the below will show in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} {code:java} 2020-11-11 08:31:57,330 ERROR [ActiveMQ BrokerService[bamboo] Task-14] [TransportConnector] Could not accept connection from tcp://192.168.1.131:41716 : javax.net.ssl.SSLException {code} The below can be observed server-side in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} when connecting to the Tomcat SSL connector or with {{-Djavax.net.debug=all}} enabled to see the issue with ActiveMQ SSL (agents JMS): {noformat} java.lang.NullPointerException at org.bouncycastle.crypto.signers.PSSSigner.generateSignature(Unknown Source) at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineSign(Unknown Source) at java.security.Signature$Delegate.engineSign(Signature.java:1382) at java.security.Signature.sign(Signature.java:698) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:181) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeProducer.produce(ECDHServerKeyExchange.java:499) at sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1020) at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:727) at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:693) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:961) at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194) at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335) at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:181) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:84) at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:74) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64) at org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1072) at org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:218) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)} {noformat} Elastic agents may show: {noformat} 2021-03-03 20:40:19,743 ERROR [tunnelserver:26224-1-thread-6] [TunnelAcceptor] Error while accepting tunnel connections. javax.net.ssl.SSLException: Received fatal alert: internal_error at sun.security.ssl.Alert.createSSLException(Alert.java:133) at sun.security.ssl.Alert.createSSLException(Alert.java:117) at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) at java.io.InputStream.read(InputStream.java:101) at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:62) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {noformat} h3. Workaround h5. *Option 1 - Rollback Java* (!) {*}NOTE{*}: For Bamboo Server only Rollback Java JDK8u272 to a previous version of JDK8 h5. *Option 2 - Disable RSASSA-PSS algorithm* (!) {*}NOTE{*}: For Bamboo Server and Elastic Agents *Bamboo Server:* * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} *or* create another file containing: {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS {code} * Add this to the {{{}$BAMBOO_INSTALL/bin/setenv.sh{}}}, in {{JVM_SUPPORT_RECOMMENDED_ARGS}} {code:java} -Djava.security.properties=/path/to/file/created.above {code} {*}Bamboo Elastic Agent:{*}{*}{*} * Please read [https://bugs.openjdk.java.net/browse/JDK-8226374|https://bugs.openjdk.java.net/browse/JDK-8226374] to understand why we need to change the Elastic Agent java settings * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS{code} This alone will fix the connection between agents and the server via the JMS broker – Bamboo Server only. If you are experiencing SSL error with HTTPS requests directly to Tomcat, {*}in addition to disabling RSASSA-PSS above{*}, you'll also need to disable TLSv1.3 by setting {{{}sslEnabledProtocols="TLSv1.2"{}}}. Example: {noformat} <Connector port="8443" maxThreads="150" minSpareThreads="25" connectionTimeout="20000" disableUploadTimeout="true" acceptCount="100" enableLookups="false" maxHttpHeaderSize="8192" useBodyEncodingForURI="true" URIEncoding="UTF-8" keystoreFile="/path/to/keystore.p12" keystorePass="changeit" scheme="https" secure="true" SSLEnabled="true" sslProtocol="TLS" sslEnabledProtocols="TLSv1.2" clientAuth="false" /> {noformat} |
New:
h3. Issue Summary
After upgrading Java to JDK 8u272 remote and elastic agents fail to connect to Bamboo server. There is often a NullPointerException during the SSL handshake. If connecting to Bamboo via a Tomcat SSL listener, the UI may fail to load with {{ERR_SSL_PROTOCOL_ERROR}} (in Chrome). This appears to be [NPE in PSSSignatureSpi#init due to unchecked null SecureRandom · Issue #633 · bcgit/bc-java · GitHub |https://github.com/bcgit/bc-java/issues/633] and fixed with the bouncy castle 1.65. Bamboo currently ships with Bouncy castle 1.64 h3. Steps to Reproduce # Using Java 8 u262 # Configure a Tomcat HTTPS listener # Configure Bamboo remote agents to use SSL: [Securing your remote agents - Bamboo Server 7.1 - Atlassian Documentation|https://confluence.atlassian.com/bamboo/securing-your-remote-agents-289277197.html] (this step is required so that the ActiveMQ broker loads BouncyCastle as a provider) ** This occurs with Bamboo's automatic JMS SSL management, not only custom certs. # Everything is working at this point # Upgrade Java on Bamboo server to JDK8 u272 and restart Bamboo h3. Expected Results Agent successfully exchanges ssl keys and connects. HTTPS connections are successful to Tomcat HTTPS listener. h3. Actual Results Agents will eventually (after they give up retrying in ~10mins of hanging at Registering with server) fail with the below in {{{}$BAMBOO_AGENT_HOME/atlassian-bamboo-agent.log{}}}: {noformat} INFO | jvm 1 | 2020/11/17 14:32:39 | 2020-11-17 03:32:39,895 TRACE [ActiveMQ Transport: ssl://local/192.168.212.131:54663] [FailoverTransport] unconnected handleTransportFailure: javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:133) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:117) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.io.DataInputStream.readInt(DataInputStream.java:387) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.lang.Thread.run(Thread.java:748) {noformat} Server-side exceptions similar to the below will show in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} {code:java} 2020-11-11 08:31:57,330 ERROR [ActiveMQ BrokerService[bamboo] Task-14] [TransportConnector] Could not accept connection from tcp://192.168.1.131:41716 : javax.net.ssl.SSLException {code} The below can be observed server-side in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} when connecting to the Tomcat SSL connector or with {{-Djavax.net.debug=all}} enabled to see the issue with ActiveMQ SSL (agents JMS): {noformat} java.lang.NullPointerException at org.bouncycastle.crypto.signers.PSSSigner.generateSignature(Unknown Source) at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineSign(Unknown Source) at java.security.Signature$Delegate.engineSign(Signature.java:1382) at java.security.Signature.sign(Signature.java:698) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:181) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeProducer.produce(ECDHServerKeyExchange.java:499) at sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1020) at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:727) at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:693) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:961) at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194) at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335) at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:181) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:84) at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:74) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64) at org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1072) at org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:218) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)} {noformat} Elastic agents may show: {noformat} 2021-03-03 20:40:19,743 ERROR [tunnelserver:26224-1-thread-6] [TunnelAcceptor] Error while accepting tunnel connections. javax.net.ssl.SSLException: Received fatal alert: internal_error at sun.security.ssl.Alert.createSSLException(Alert.java:133) at sun.security.ssl.Alert.createSSLException(Alert.java:117) at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) at java.io.InputStream.read(InputStream.java:101) at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:62) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {noformat} h3. Workaround h5. *Option 1 - Rollback Java* (!) {*}NOTE{*}: For Bamboo Server only Rollback Java JDK8u272 to a previous version of JDK8 h5. *Option 2 - Disable RSASSA-PSS algorithm* (!) {*}NOTE{*}: For Bamboo Server and Elastic Agents *Bamboo Server:* * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} *or* create another file containing: {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS {code} * Add this to the {{{}$BAMBOO_INSTALL/bin/setenv.sh{}}}, in {{JVM_SUPPORT_RECOMMENDED_ARGS}} {code:java} -Djava.security.properties=/path/to/file/created.above {code} {*}Bamboo Elastic Agent:{*} * Please read [https://bugs.openjdk.java.net/browse/JDK-8226374|https://bugs.openjdk.java.net/browse/JDK-8226374] to understand why we need to change the Elastic Agent java settings * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS{code} This alone will fix the connection between agents and the server via the JMS broker – Bamboo Server only. If you are experiencing SSL error with HTTPS requests directly to Tomcat, {*}in addition to disabling RSASSA-PSS above{*}, you'll also need to disable TLSv1.3 by setting {{{}sslEnabledProtocols="TLSv1.2"{}}}. Example: {noformat} <Connector port="8443" maxThreads="150" minSpareThreads="25" connectionTimeout="20000" disableUploadTimeout="true" acceptCount="100" enableLookups="false" maxHttpHeaderSize="8192" useBodyEncodingForURI="true" URIEncoding="UTF-8" keystoreFile="/path/to/keystore.p12" keystorePass="changeit" scheme="https" secure="true" SSLEnabled="true" sslProtocol="TLS" sslEnabledProtocols="TLSv1.2" clientAuth="false" /> {noformat} |
| Description |
Original:
h3. Issue Summary
After upgrading Java to JDK 8u272 remote and elastic agents fail to connect to Bamboo server. There is often a NullPointerException during the SSL handshake. If connecting to Bamboo via a Tomcat SSL listener, the UI may fail to load with {{ERR_SSL_PROTOCOL_ERROR}} (in Chrome). This appears to be [NPE in PSSSignatureSpi#init due to unchecked null SecureRandom · Issue #633 · bcgit/bc-java · GitHub |https://github.com/bcgit/bc-java/issues/633] and fixed with the bouncy castle 1.65. Bamboo currently ships with Bouncy castle 1.64 h3. Steps to Reproduce # Using Java 8 u262 # Configure a Tomcat HTTPS listener # Configure Bamboo remote agents to use SSL: [Securing your remote agents - Bamboo Server 7.1 - Atlassian Documentation|https://confluence.atlassian.com/bamboo/securing-your-remote-agents-289277197.html] (this step is required so that the ActiveMQ broker loads BouncyCastle as a provider) ** This occurs with Bamboo's automatic JMS SSL management, not only custom certs. # Everything is working at this point # Upgrade Java on Bamboo server to JDK8 u272 and restart Bamboo h3. Expected Results Agent successfully exchanges ssl keys and connects. HTTPS connections are successful to Tomcat HTTPS listener. h3. Actual Results Agents will eventually (after they give up retrying in ~10mins of hanging at Registering with server) fail with the below in {{{}$BAMBOO_AGENT_HOME/atlassian-bamboo-agent.log{}}}: {noformat} INFO | jvm 1 | 2020/11/17 14:32:39 | 2020-11-17 03:32:39,895 TRACE [ActiveMQ Transport: ssl://local/192.168.212.131:54663] [FailoverTransport] unconnected handleTransportFailure: javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:133) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:117) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.io.DataInputStream.readInt(DataInputStream.java:387) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.lang.Thread.run(Thread.java:748) {noformat} Server-side exceptions similar to the below will show in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} {code:java} 2020-11-11 08:31:57,330 ERROR [ActiveMQ BrokerService[bamboo] Task-14] [TransportConnector] Could not accept connection from tcp://192.168.1.131:41716 : javax.net.ssl.SSLException {code} The below can be observed server-side in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} when connecting to the Tomcat SSL connector or with {{-Djavax.net.debug=all}} enabled to see the issue with ActiveMQ SSL (agents JMS): {noformat} java.lang.NullPointerException at org.bouncycastle.crypto.signers.PSSSigner.generateSignature(Unknown Source) at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineSign(Unknown Source) at java.security.Signature$Delegate.engineSign(Signature.java:1382) at java.security.Signature.sign(Signature.java:698) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:181) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeProducer.produce(ECDHServerKeyExchange.java:499) at sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1020) at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:727) at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:693) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:961) at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194) at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335) at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:181) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:84) at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:74) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64) at org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1072) at org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:218) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)} {noformat} Elastic agents may show: {noformat} 2021-03-03 20:40:19,743 ERROR [tunnelserver:26224-1-thread-6] [TunnelAcceptor] Error while accepting tunnel connections. javax.net.ssl.SSLException: Received fatal alert: internal_error at sun.security.ssl.Alert.createSSLException(Alert.java:133) at sun.security.ssl.Alert.createSSLException(Alert.java:117) at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) at java.io.InputStream.read(InputStream.java:101) at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:62) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {noformat} h3. Workaround h5. *Option 1 - Rollback Java* (!) {*}NOTE{*}: For Bamboo Server only Rollback Java JDK8u272 to a previous version of JDK8 h5. *Option 2 - Disable RSASSA-PSS algorithm* (!) {*}NOTE{*}: For Bamboo Server and Elastic Agents *Bamboo Server:* * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} *or* create another file containing: {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS {code} * Add this to the {{{}$BAMBOO_INSTALL/bin/setenv.sh{}}}, in {{JVM_SUPPORT_RECOMMENDED_ARGS}} {code:java} -Djava.security.properties=/path/to/file/created.above {code} *Bamboo Elastic Agent:* * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS{code} This alone will fix the connection between agents and the server via the JMS broker – Bamboo Server only. If you are experiencing SSL error with HTTPS requests directly to Tomcat, {*}in addition to disabling RSASSA-PSS above{*}, you'll also need to disable TLSv1.3 by setting {{{}sslEnabledProtocols="TLSv1.2"{}}}. Example: {noformat} <Connector port="8443" maxThreads="150" minSpareThreads="25" connectionTimeout="20000" disableUploadTimeout="true" acceptCount="100" enableLookups="false" maxHttpHeaderSize="8192" useBodyEncodingForURI="true" URIEncoding="UTF-8" keystoreFile="/path/to/keystore.p12" keystorePass="changeit" scheme="https" secure="true" SSLEnabled="true" sslProtocol="TLS" sslEnabledProtocols="TLSv1.2" clientAuth="false" /> {noformat} |
New:
h3. Issue Summary
After upgrading Java to JDK 8u272 remote and elastic agents fail to connect to Bamboo server. There is often a NullPointerException during the SSL handshake. If connecting to Bamboo via a Tomcat SSL listener, the UI may fail to load with {{ERR_SSL_PROTOCOL_ERROR}} (in Chrome). This appears to be [NPE in PSSSignatureSpi#init due to unchecked null SecureRandom · Issue #633 · bcgit/bc-java · GitHub |https://github.com/bcgit/bc-java/issues/633] and fixed with the bouncy castle 1.65. Bamboo currently ships with Bouncy castle 1.64 h3. Steps to Reproduce # Using Java 8 u262 # Configure a Tomcat HTTPS listener # Configure Bamboo remote agents to use SSL: [Securing your remote agents - Bamboo Server 7.1 - Atlassian Documentation|https://confluence.atlassian.com/bamboo/securing-your-remote-agents-289277197.html] (this step is required so that the ActiveMQ broker loads BouncyCastle as a provider) ** This occurs with Bamboo's automatic JMS SSL management, not only custom certs. # Everything is working at this point # Upgrade Java on Bamboo server to JDK8 u272 and restart Bamboo h3. Expected Results Agent successfully exchanges ssl keys and connects. HTTPS connections are successful to Tomcat HTTPS listener. h3. Actual Results Agents will eventually (after they give up retrying in ~10mins of hanging at Registering with server) fail with the below in {{{}$BAMBOO_AGENT_HOME/atlassian-bamboo-agent.log{}}}: {noformat} INFO | jvm 1 | 2020/11/17 14:32:39 | 2020-11-17 03:32:39,895 TRACE [ActiveMQ Transport: ssl://local/192.168.212.131:54663] [FailoverTransport] unconnected handleTransportFailure: javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:133) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:117) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.io.DataInputStream.readInt(DataInputStream.java:387) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.lang.Thread.run(Thread.java:748) {noformat} Server-side exceptions similar to the below will show in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} {code:java} 2020-11-11 08:31:57,330 ERROR [ActiveMQ BrokerService[bamboo] Task-14] [TransportConnector] Could not accept connection from tcp://192.168.1.131:41716 : javax.net.ssl.SSLException {code} The below can be observed server-side in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} when connecting to the Tomcat SSL connector or with {{-Djavax.net.debug=all}} enabled to see the issue with ActiveMQ SSL (agents JMS): {noformat} java.lang.NullPointerException at org.bouncycastle.crypto.signers.PSSSigner.generateSignature(Unknown Source) at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineSign(Unknown Source) at java.security.Signature$Delegate.engineSign(Signature.java:1382) at java.security.Signature.sign(Signature.java:698) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:181) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeProducer.produce(ECDHServerKeyExchange.java:499) at sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1020) at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:727) at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:693) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:961) at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194) at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335) at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:181) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:84) at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:74) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64) at org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1072) at org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:218) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)} {noformat} Elastic agents may show: {noformat} 2021-03-03 20:40:19,743 ERROR [tunnelserver:26224-1-thread-6] [TunnelAcceptor] Error while accepting tunnel connections. javax.net.ssl.SSLException: Received fatal alert: internal_error at sun.security.ssl.Alert.createSSLException(Alert.java:133) at sun.security.ssl.Alert.createSSLException(Alert.java:117) at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) at java.io.InputStream.read(InputStream.java:101) at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:62) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {noformat} h3. Workaround h5. *Option 1 - Rollback Java* (!) {*}NOTE{*}: For Bamboo Server only Rollback Java JDK8u272 to a previous version of JDK8 h5. *Option 2 - Disable RSASSA-PSS algorithm* (!) {*}NOTE{*}: For Bamboo Server and Elastic Agents *Bamboo Server:* * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} *or* create another file containing: {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS {code} * Add this to the {{{}$BAMBOO_INSTALL/bin/setenv.sh{}}}, in {{JVM_SUPPORT_RECOMMENDED_ARGS}} {code:java} -Djava.security.properties=/path/to/file/created.above {code} {*}Bamboo Elastic Agent:{*}{*}{*} * Please read [https://bugs.openjdk.java.net/browse/JDK-8226374|https://bugs.openjdk.java.net/browse/JDK-8226374] to understand why we need to change the Elastic Agent java settings * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS{code} This alone will fix the connection between agents and the server via the JMS broker – Bamboo Server only. If you are experiencing SSL error with HTTPS requests directly to Tomcat, {*}in addition to disabling RSASSA-PSS above{*}, you'll also need to disable TLSv1.3 by setting {{{}sslEnabledProtocols="TLSv1.2"{}}}. Example: {noformat} <Connector port="8443" maxThreads="150" minSpareThreads="25" connectionTimeout="20000" disableUploadTimeout="true" acceptCount="100" enableLookups="false" maxHttpHeaderSize="8192" useBodyEncodingForURI="true" URIEncoding="UTF-8" keystoreFile="/path/to/keystore.p12" keystorePass="changeit" scheme="https" secure="true" SSLEnabled="true" sslProtocol="TLS" sslEnabledProtocols="TLSv1.2" clientAuth="false" /> {noformat} |
| Description |
Original:
h3. Issue Summary
After upgrading Java to JDK 8u272 remote and elastic agents fail to connect to Bamboo server. There is often a NullPointerException during the SSL handshake. If connecting to Bamboo via a Tomcat SSL listener, the UI may fail to load with {{ERR_SSL_PROTOCOL_ERROR}} (in Chrome). This appears to be [NPE in PSSSignatureSpi#init due to unchecked null SecureRandom · Issue #633 · bcgit/bc-java · GitHub |https://github.com/bcgit/bc-java/issues/633] and fixed with the bouncy castle 1.65. Bamboo currently ships with Bouncy castle 1.64 h3. Steps to Reproduce # Using Java 8 u262 # Configure a Tomcat HTTPS listener # Configure Bamboo remote agents to use SSL: [Securing your remote agents - Bamboo Server 7.1 - Atlassian Documentation|https://confluence.atlassian.com/bamboo/securing-your-remote-agents-289277197.html] (this step is required so that the ActiveMQ broker loads BouncyCastle as a provider) ** This occurs with Bamboo's automatic JMS SSL management, not only custom certs. # Everything is working at this point # Upgrade Java on Bamboo server to JDK8 u272 and restart Bamboo h3. Expected Results Agent successfully exchanges ssl keys and connects. HTTPS connections are successful to Tomcat HTTPS listener. h3. Actual Results Agents will eventually (after they give up retrying in ~10mins of hanging at Registering with server) fail with the below in {{{}$BAMBOO_AGENT_HOME/atlassian-bamboo-agent.log{}}}: {noformat} INFO | jvm 1 | 2020/11/17 14:32:39 | 2020-11-17 03:32:39,895 TRACE [ActiveMQ Transport: ssl://local/192.168.212.131:54663] [FailoverTransport] unconnected handleTransportFailure: javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:133) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:117) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.io.DataInputStream.readInt(DataInputStream.java:387) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.lang.Thread.run(Thread.java:748) {noformat} Server-side exceptions similar to the below will show in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} {code:java} 2020-11-11 08:31:57,330 ERROR [ActiveMQ BrokerService[bamboo] Task-14] [TransportConnector] Could not accept connection from tcp://192.168.1.131:41716 : javax.net.ssl.SSLException {code} The below can be observed server-side in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} when connecting to the Tomcat SSL connector or with {{-Djavax.net.debug=all}} enabled to see the issue with ActiveMQ SSL (agents JMS): {noformat} java.lang.NullPointerException at org.bouncycastle.crypto.signers.PSSSigner.generateSignature(Unknown Source) at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineSign(Unknown Source) at java.security.Signature$Delegate.engineSign(Signature.java:1382) at java.security.Signature.sign(Signature.java:698) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:181) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeProducer.produce(ECDHServerKeyExchange.java:499) at sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1020) at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:727) at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:693) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:961) at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194) at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335) at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:181) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:84) at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:74) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64) at org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1072) at org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:218) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)} {noformat} Elastic agents may show: {noformat} 2021-03-03 20:40:19,743 ERROR [tunnelserver:26224-1-thread-6] [TunnelAcceptor] Error while accepting tunnel connections. javax.net.ssl.SSLException: Received fatal alert: internal_error at sun.security.ssl.Alert.createSSLException(Alert.java:133) at sun.security.ssl.Alert.createSSLException(Alert.java:117) at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) at java.io.InputStream.read(InputStream.java:101) at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:62) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {noformat} h3. Workaround (i) The below workaround is performed on the Bamboo server side (not agent) h4. Option 1 - Rollback Java Rollback Java JDK8u272 to a previous version of JDK8 h4. Option 2 - Disable RSASSA-PSS algorithm (!) {*}NOTE{*}: This workaround applies to Bamboo Server and Elastic Agents *Bamboo Server:* * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} *or* create another file containing: {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS {code} * Add this to the {{{}$BAMBOO_INSTALL/bin/setenv.sh{}}}, in {{JVM_SUPPORT_RECOMMENDED_ARGS}} {code:java} -Djava.security.properties=/path/to/file/created.above {code} *Bamboo Elastic Agent:* * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS{code} This alone will fix the connection between agents and the server via the JMS broker – Bamboo Server only. If you are experiencing SSL error with HTTPS requests directly to Tomcat, {*}in addition to disabling RSASSA-PSS above{*}, you'll also need to disable TLSv1.3 by setting {{{}sslEnabledProtocols="TLSv1.2"{}}}. Example: {noformat} <Connector port="8443" maxThreads="150" minSpareThreads="25" connectionTimeout="20000" disableUploadTimeout="true" acceptCount="100" enableLookups="false" maxHttpHeaderSize="8192" useBodyEncodingForURI="true" URIEncoding="UTF-8" keystoreFile="/path/to/keystore.p12" keystorePass="changeit" scheme="https" secure="true" SSLEnabled="true" sslProtocol="TLS" sslEnabledProtocols="TLSv1.2" clientAuth="false" /> {noformat} |
New:
h3. Issue Summary
After upgrading Java to JDK 8u272 remote and elastic agents fail to connect to Bamboo server. There is often a NullPointerException during the SSL handshake. If connecting to Bamboo via a Tomcat SSL listener, the UI may fail to load with {{ERR_SSL_PROTOCOL_ERROR}} (in Chrome). This appears to be [NPE in PSSSignatureSpi#init due to unchecked null SecureRandom · Issue #633 · bcgit/bc-java · GitHub |https://github.com/bcgit/bc-java/issues/633] and fixed with the bouncy castle 1.65. Bamboo currently ships with Bouncy castle 1.64 h3. Steps to Reproduce # Using Java 8 u262 # Configure a Tomcat HTTPS listener # Configure Bamboo remote agents to use SSL: [Securing your remote agents - Bamboo Server 7.1 - Atlassian Documentation|https://confluence.atlassian.com/bamboo/securing-your-remote-agents-289277197.html] (this step is required so that the ActiveMQ broker loads BouncyCastle as a provider) ** This occurs with Bamboo's automatic JMS SSL management, not only custom certs. # Everything is working at this point # Upgrade Java on Bamboo server to JDK8 u272 and restart Bamboo h3. Expected Results Agent successfully exchanges ssl keys and connects. HTTPS connections are successful to Tomcat HTTPS listener. h3. Actual Results Agents will eventually (after they give up retrying in ~10mins of hanging at Registering with server) fail with the below in {{{}$BAMBOO_AGENT_HOME/atlassian-bamboo-agent.log{}}}: {noformat} INFO | jvm 1 | 2020/11/17 14:32:39 | 2020-11-17 03:32:39,895 TRACE [ActiveMQ Transport: ssl://local/192.168.212.131:54663] [FailoverTransport] unconnected handleTransportFailure: javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:133) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:117) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.io.DataInputStream.readInt(DataInputStream.java:387) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.lang.Thread.run(Thread.java:748) {noformat} Server-side exceptions similar to the below will show in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} {code:java} 2020-11-11 08:31:57,330 ERROR [ActiveMQ BrokerService[bamboo] Task-14] [TransportConnector] Could not accept connection from tcp://192.168.1.131:41716 : javax.net.ssl.SSLException {code} The below can be observed server-side in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} when connecting to the Tomcat SSL connector or with {{-Djavax.net.debug=all}} enabled to see the issue with ActiveMQ SSL (agents JMS): {noformat} java.lang.NullPointerException at org.bouncycastle.crypto.signers.PSSSigner.generateSignature(Unknown Source) at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineSign(Unknown Source) at java.security.Signature$Delegate.engineSign(Signature.java:1382) at java.security.Signature.sign(Signature.java:698) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:181) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeProducer.produce(ECDHServerKeyExchange.java:499) at sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1020) at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:727) at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:693) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:961) at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194) at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335) at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:181) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:84) at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:74) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64) at org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1072) at org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:218) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)} {noformat} Elastic agents may show: {noformat} 2021-03-03 20:40:19,743 ERROR [tunnelserver:26224-1-thread-6] [TunnelAcceptor] Error while accepting tunnel connections. javax.net.ssl.SSLException: Received fatal alert: internal_error at sun.security.ssl.Alert.createSSLException(Alert.java:133) at sun.security.ssl.Alert.createSSLException(Alert.java:117) at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) at java.io.InputStream.read(InputStream.java:101) at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:62) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {noformat} h3. Workaround h5. *Option 1 - Rollback Java* (!) {*}NOTE{*}: For Bamboo Server only Rollback Java JDK8u272 to a previous version of JDK8 h5. *Option 2 - Disable RSASSA-PSS algorithm* (!) {*}NOTE{*}: For Bamboo Server and Elastic Agents *Bamboo Server:* * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} *or* create another file containing: {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS {code} * Add this to the {{{}$BAMBOO_INSTALL/bin/setenv.sh{}}}, in {{JVM_SUPPORT_RECOMMENDED_ARGS}} {code:java} -Djava.security.properties=/path/to/file/created.above {code} *Bamboo Elastic Agent:* * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS{code} This alone will fix the connection between agents and the server via the JMS broker – Bamboo Server only. If you are experiencing SSL error with HTTPS requests directly to Tomcat, {*}in addition to disabling RSASSA-PSS above{*}, you'll also need to disable TLSv1.3 by setting {{{}sslEnabledProtocols="TLSv1.2"{}}}. Example: {noformat} <Connector port="8443" maxThreads="150" minSpareThreads="25" connectionTimeout="20000" disableUploadTimeout="true" acceptCount="100" enableLookups="false" maxHttpHeaderSize="8192" useBodyEncodingForURI="true" URIEncoding="UTF-8" keystoreFile="/path/to/keystore.p12" keystorePass="changeit" scheme="https" secure="true" SSLEnabled="true" sslProtocol="TLS" sslEnabledProtocols="TLSv1.2" clientAuth="false" /> {noformat} |
| Description |
Original:
h3. Issue Summary
After upgrading Java to JDK 8u272 remote and elastic agents fail to connect to Bamboo server. There is often a NullPointerException during the SSL handshake. If connecting to Bamboo via a Tomcat SSL listener, the UI may fail to load with {{ERR_SSL_PROTOCOL_ERROR}} (in Chrome). This appears to be [NPE in PSSSignatureSpi#init due to unchecked null SecureRandom · Issue #633 · bcgit/bc-java · GitHub |https://github.com/bcgit/bc-java/issues/633] and fixed with the bouncy castle 1.65. Bamboo currently ships with Bouncy castle 1.64 h3. Steps to Reproduce # Using Java 8 u262 # Configure a Tomcat HTTPS listener # Configure Bamboo remote agents to use SSL: [Securing your remote agents - Bamboo Server 7.1 - Atlassian Documentation|https://confluence.atlassian.com/bamboo/securing-your-remote-agents-289277197.html] (this step is required so that the ActiveMQ broker loads BouncyCastle as a provider) ** This occurs with Bamboo's automatic JMS SSL management, not only custom certs. # Everything is working at this point # Upgrade Java on Bamboo server to JDK8 u272 and restart Bamboo h3. Expected Results Agent successfully exchanges ssl keys and connects. HTTPS connections are successful to Tomcat HTTPS listener. h3. Actual Results Agents will eventually (after they give up retrying in ~10mins of hanging at Registering with server) fail with the below in {{{}$BAMBOO_AGENT_HOME/atlassian-bamboo-agent.log{}}}: {noformat} INFO | jvm 1 | 2020/11/17 14:32:39 | 2020-11-17 03:32:39,895 TRACE [ActiveMQ Transport: ssl://local/192.168.212.131:54663] [FailoverTransport] unconnected handleTransportFailure: javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:133) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:117) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.io.DataInputStream.readInt(DataInputStream.java:387) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.lang.Thread.run(Thread.java:748) {noformat} Server-side exceptions similar to the below will show in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} {code:java} 2020-11-11 08:31:57,330 ERROR [ActiveMQ BrokerService[bamboo] Task-14] [TransportConnector] Could not accept connection from tcp://192.168.1.131:41716 : javax.net.ssl.SSLException {code} The below can be observed server-side in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} when connecting to the Tomcat SSL connector or with {{-Djavax.net.debug=all}} enabled to see the issue with ActiveMQ SSL (agents JMS): {noformat} java.lang.NullPointerException at org.bouncycastle.crypto.signers.PSSSigner.generateSignature(Unknown Source) at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineSign(Unknown Source) at java.security.Signature$Delegate.engineSign(Signature.java:1382) at java.security.Signature.sign(Signature.java:698) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:181) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeProducer.produce(ECDHServerKeyExchange.java:499) at sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1020) at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:727) at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:693) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:961) at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194) at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335) at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:181) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:84) at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:74) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64) at org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1072) at org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:218) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)} {noformat} Elastic agents may show: {noformat} 2021-03-03 20:40:19,743 ERROR [tunnelserver:26224-1-thread-6] [TunnelAcceptor] Error while accepting tunnel connections. javax.net.ssl.SSLException: Received fatal alert: internal_error at sun.security.ssl.Alert.createSSLException(Alert.java:133) at sun.security.ssl.Alert.createSSLException(Alert.java:117) at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) at java.io.InputStream.read(InputStream.java:101) at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:62) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {noformat} h3. Workaround (i) The below workaround is performed on the Bamboo server side (not agent) h4. Option 1 - Rollback Java Rollback Java JDK8u272 to a previous version of JDK8 h4. Option 2 - Disable specific Java algorithms (!) {*}NOTE{*}: This workaround applies to Bamboo Server and Elastic Agents *Disable RSASSA-PSS:* * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} *or* create another file containing: {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS {code} * Add this to the {{{}$BAMBOO_INSTALL/bin/setenv.sh{}}}, in {{{}JVM_SUPPORT_RECOMMENDED_ARGS{}}}: {code:java} -Djava.security.properties=/path/to/file/created.above {code} This alone will fix the connection between agents and the server via the JMS broker – Bamboo Server only. If you are experiencing SSL error with HTTPS requests directly to Tomcat, {*}in addition to disabling RSASSA-PSS above{*}, you'll also need to disable TLSv1.3 by setting {{{}sslEnabledProtocols="TLSv1.2"{}}}. Example: {noformat} <Connector port="8443" maxThreads="150" minSpareThreads="25" connectionTimeout="20000" disableUploadTimeout="true" acceptCount="100" enableLookups="false" maxHttpHeaderSize="8192" useBodyEncodingForURI="true" URIEncoding="UTF-8" keystoreFile="/path/to/keystore.p12" keystorePass="changeit" scheme="https" secure="true" SSLEnabled="true" sslProtocol="TLS" sslEnabledProtocols="TLSv1.2" clientAuth="false" /> {noformat} |
New:
h3. Issue Summary
After upgrading Java to JDK 8u272 remote and elastic agents fail to connect to Bamboo server. There is often a NullPointerException during the SSL handshake. If connecting to Bamboo via a Tomcat SSL listener, the UI may fail to load with {{ERR_SSL_PROTOCOL_ERROR}} (in Chrome). This appears to be [NPE in PSSSignatureSpi#init due to unchecked null SecureRandom · Issue #633 · bcgit/bc-java · GitHub |https://github.com/bcgit/bc-java/issues/633] and fixed with the bouncy castle 1.65. Bamboo currently ships with Bouncy castle 1.64 h3. Steps to Reproduce # Using Java 8 u262 # Configure a Tomcat HTTPS listener # Configure Bamboo remote agents to use SSL: [Securing your remote agents - Bamboo Server 7.1 - Atlassian Documentation|https://confluence.atlassian.com/bamboo/securing-your-remote-agents-289277197.html] (this step is required so that the ActiveMQ broker loads BouncyCastle as a provider) ** This occurs with Bamboo's automatic JMS SSL management, not only custom certs. # Everything is working at this point # Upgrade Java on Bamboo server to JDK8 u272 and restart Bamboo h3. Expected Results Agent successfully exchanges ssl keys and connects. HTTPS connections are successful to Tomcat HTTPS listener. h3. Actual Results Agents will eventually (after they give up retrying in ~10mins of hanging at Registering with server) fail with the below in {{{}$BAMBOO_AGENT_HOME/atlassian-bamboo-agent.log{}}}: {noformat} INFO | jvm 1 | 2020/11/17 14:32:39 | 2020-11-17 03:32:39,895 TRACE [ActiveMQ Transport: ssl://local/192.168.212.131:54663] [FailoverTransport] unconnected handleTransportFailure: javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:133) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:117) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.io.DataInputStream.readInt(DataInputStream.java:387) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.lang.Thread.run(Thread.java:748) {noformat} Server-side exceptions similar to the below will show in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} {code:java} 2020-11-11 08:31:57,330 ERROR [ActiveMQ BrokerService[bamboo] Task-14] [TransportConnector] Could not accept connection from tcp://192.168.1.131:41716 : javax.net.ssl.SSLException {code} The below can be observed server-side in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} when connecting to the Tomcat SSL connector or with {{-Djavax.net.debug=all}} enabled to see the issue with ActiveMQ SSL (agents JMS): {noformat} java.lang.NullPointerException at org.bouncycastle.crypto.signers.PSSSigner.generateSignature(Unknown Source) at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineSign(Unknown Source) at java.security.Signature$Delegate.engineSign(Signature.java:1382) at java.security.Signature.sign(Signature.java:698) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:181) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeProducer.produce(ECDHServerKeyExchange.java:499) at sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1020) at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:727) at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:693) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:961) at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194) at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335) at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:181) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:84) at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:74) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64) at org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1072) at org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:218) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)} {noformat} Elastic agents may show: {noformat} 2021-03-03 20:40:19,743 ERROR [tunnelserver:26224-1-thread-6] [TunnelAcceptor] Error while accepting tunnel connections. javax.net.ssl.SSLException: Received fatal alert: internal_error at sun.security.ssl.Alert.createSSLException(Alert.java:133) at sun.security.ssl.Alert.createSSLException(Alert.java:117) at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) at java.io.InputStream.read(InputStream.java:101) at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:62) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {noformat} h3. Workaround (i) The below workaround is performed on the Bamboo server side (not agent) h4. Option 1 - Rollback Java Rollback Java JDK8u272 to a previous version of JDK8 h4. Option 2 - Disable RSASSA-PSS algorithm (!) {*}NOTE{*}: This workaround applies to Bamboo Server and Elastic Agents *Bamboo Server:* * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} *or* create another file containing: {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS {code} * Add this to the {{{}$BAMBOO_INSTALL/bin/setenv.sh{}}}, in {{JVM_SUPPORT_RECOMMENDED_ARGS}} {code:java} -Djava.security.properties=/path/to/file/created.above {code} *Bamboo Elastic Agent:* * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS{code} This alone will fix the connection between agents and the server via the JMS broker – Bamboo Server only. If you are experiencing SSL error with HTTPS requests directly to Tomcat, {*}in addition to disabling RSASSA-PSS above{*}, you'll also need to disable TLSv1.3 by setting {{{}sslEnabledProtocols="TLSv1.2"{}}}. Example: {noformat} <Connector port="8443" maxThreads="150" minSpareThreads="25" connectionTimeout="20000" disableUploadTimeout="true" acceptCount="100" enableLookups="false" maxHttpHeaderSize="8192" useBodyEncodingForURI="true" URIEncoding="UTF-8" keystoreFile="/path/to/keystore.p12" keystorePass="changeit" scheme="https" secure="true" SSLEnabled="true" sslProtocol="TLS" sslEnabledProtocols="TLSv1.2" clientAuth="false" /> {noformat} |
| Description |
Original:
h3. Issue Summary
After upgrading Java to JDK 8u272 remote and elastic agents fail to connect to Bamboo server. There is often a NullPointerException during the SSL handshake. If connecting to Bamboo via a Tomcat SSL listener, the UI may fail to load with {{ERR_SSL_PROTOCOL_ERROR}} (in Chrome). This appears to be [NPE in PSSSignatureSpi#init due to unchecked null SecureRandom · Issue #633 · bcgit/bc-java · GitHub |https://github.com/bcgit/bc-java/issues/633] and fixed with the bouncy castle 1.65. Bamboo currently ships with Bouncy castle 1.64 h3. Steps to Reproduce # Using Java 8 u262 # Configure a Tomcat HTTPS listener # Configure Bamboo remote agents to use SSL: [Securing your remote agents - Bamboo Server 7.1 - Atlassian Documentation|https://confluence.atlassian.com/bamboo/securing-your-remote-agents-289277197.html] (this step is required so that the ActiveMQ broker loads BouncyCastle as a provider) ** This occurs with Bamboo's automatic JMS SSL management, not only custom certs. # Everything is working at this point # Upgrade Java on Bamboo server to JDK8 u272 and restart Bamboo h3. Expected Results Agent successfully exchanges ssl keys and connects. HTTPS connections are successful to Tomcat HTTPS listener. h3. Actual Results Agents will eventually (after they give up retrying in ~10mins of hanging at Registering with server) fail with the below in {{$BAMBOO_AGENT_HOME/atlassian-bamboo-agent.log}}: {noformat} INFO | jvm 1 | 2020/11/17 14:32:39 | 2020-11-17 03:32:39,895 TRACE [ActiveMQ Transport: ssl://local/192.168.212.131:54663] [FailoverTransport] unconnected handleTransportFailure: javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:133) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:117) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.io.DataInputStream.readInt(DataInputStream.java:387) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.lang.Thread.run(Thread.java:748) {noformat} Server-side exceptions similar to the below will show in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} {code:java} 2020-11-11 08:31:57,330 ERROR [ActiveMQ BrokerService[bamboo] Task-14] [TransportConnector] Could not accept connection from tcp://192.168.1.131:41716 : javax.net.ssl.SSLException {code} The below can be observed server-side in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} when connecting to the Tomcat SSL connector or with {{-Djavax.net.debug=all}} enabled to see the issue with ActiveMQ SSL (agents JMS): {noformat} java.lang.NullPointerException at org.bouncycastle.crypto.signers.PSSSigner.generateSignature(Unknown Source) at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineSign(Unknown Source) at java.security.Signature$Delegate.engineSign(Signature.java:1382) at java.security.Signature.sign(Signature.java:698) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:181) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeProducer.produce(ECDHServerKeyExchange.java:499) at sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1020) at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:727) at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:693) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:961) at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194) at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335) at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:181) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:84) at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:74) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64) at org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1072) at org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:218) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)} {noformat} Elastic agents may show: {noformat} 2021-03-03 20:40:19,743 ERROR [tunnelserver:26224-1-thread-6] [TunnelAcceptor] Error while accepting tunnel connections. javax.net.ssl.SSLException: Received fatal alert: internal_error at sun.security.ssl.Alert.createSSLException(Alert.java:133) at sun.security.ssl.Alert.createSSLException(Alert.java:117) at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) at java.io.InputStream.read(InputStream.java:101) at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:62) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {noformat} h3. Workaround (i) The below workaround is performed on the Bamboo server side (not agent) h6. Option 1 Rollback Java JDK8u272 to a previous version of JDK8 h6. Option 2 Disable RSASSA-PSS: Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} *or* create another file containing: {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS {code} and add this to the {{$BAMBOO_INSTALL/bin/setenv.sh}}, in {{JVM_SUPPORT_RECOMMENDED_ARGS}}: {code:java} -Djava.security.properties=/path/to/file/created.above {code} This alone will fix the connection between agents and the server via the JMS broker. If you are experiencing SSL error with HTTPS requests directly to Tomcat, *in addition to disabling RSASSA-PSS above*, you'll also need to disable TLSv1.3 by setting {{sslEnabledProtocols="TLSv1.2"}}. Example: {noformat} <Connector port="8443" maxThreads="150" minSpareThreads="25" connectionTimeout="20000" disableUploadTimeout="true" acceptCount="100" enableLookups="false" maxHttpHeaderSize="8192" useBodyEncodingForURI="true" URIEncoding="UTF-8" keystoreFile="/path/to/keystore.p12" keystorePass="changeit" scheme="https" secure="true" SSLEnabled="true" sslProtocol="TLS" sslEnabledProtocols="TLSv1.2" clientAuth="false" /> {noformat} |
New:
h3. Issue Summary
After upgrading Java to JDK 8u272 remote and elastic agents fail to connect to Bamboo server. There is often a NullPointerException during the SSL handshake. If connecting to Bamboo via a Tomcat SSL listener, the UI may fail to load with {{ERR_SSL_PROTOCOL_ERROR}} (in Chrome). This appears to be [NPE in PSSSignatureSpi#init due to unchecked null SecureRandom · Issue #633 · bcgit/bc-java · GitHub |https://github.com/bcgit/bc-java/issues/633] and fixed with the bouncy castle 1.65. Bamboo currently ships with Bouncy castle 1.64 h3. Steps to Reproduce # Using Java 8 u262 # Configure a Tomcat HTTPS listener # Configure Bamboo remote agents to use SSL: [Securing your remote agents - Bamboo Server 7.1 - Atlassian Documentation|https://confluence.atlassian.com/bamboo/securing-your-remote-agents-289277197.html] (this step is required so that the ActiveMQ broker loads BouncyCastle as a provider) ** This occurs with Bamboo's automatic JMS SSL management, not only custom certs. # Everything is working at this point # Upgrade Java on Bamboo server to JDK8 u272 and restart Bamboo h3. Expected Results Agent successfully exchanges ssl keys and connects. HTTPS connections are successful to Tomcat HTTPS listener. h3. Actual Results Agents will eventually (after they give up retrying in ~10mins of hanging at Registering with server) fail with the below in {{{}$BAMBOO_AGENT_HOME/atlassian-bamboo-agent.log{}}}: {noformat} INFO | jvm 1 | 2020/11/17 14:32:39 | 2020-11-17 03:32:39,895 TRACE [ActiveMQ Transport: ssl://local/192.168.212.131:54663] [FailoverTransport] unconnected handleTransportFailure: javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | javax.net.ssl.SSLException: Received fatal alert: internal_error INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:133) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert.createSSLException(Alert.java:117) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) INFO | jvm 1 | 2020/11/17 14:32:39 | at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.fill(TcpBufferedInputStream.java:50) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.fill(TcpTransport.java:634) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpBufferedInputStream.read(TcpBufferedInputStream.java:59) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport$2.read(TcpTransport.java:619) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.io.DataInputStream.readInt(DataInputStream.java:387) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.openwire.OpenWireFormat.unmarshal(OpenWireFormat.java:268) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.readCommand(TcpTransport.java:240) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:232) INFO | jvm 1 | 2020/11/17 14:32:39 | at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:215) INFO | jvm 1 | 2020/11/17 14:32:39 | at java.lang.Thread.run(Thread.java:748) {noformat} Server-side exceptions similar to the below will show in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} {code:java} 2020-11-11 08:31:57,330 ERROR [ActiveMQ BrokerService[bamboo] Task-14] [TransportConnector] Could not accept connection from tcp://192.168.1.131:41716 : javax.net.ssl.SSLException {code} The below can be observed server-side in {{$BAMBOO_HOME/logs/atlassian-bamboo.log}} when connecting to the Tomcat SSL connector or with {{-Djavax.net.debug=all}} enabled to see the issue with ActiveMQ SSL (agents JMS): {noformat} java.lang.NullPointerException at org.bouncycastle.crypto.signers.PSSSigner.generateSignature(Unknown Source) at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineSign(Unknown Source) at java.security.Signature$Delegate.engineSign(Signature.java:1382) at java.security.Signature.sign(Signature.java:698) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:181) at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeProducer.produce(ECDHServerKeyExchange.java:499) at sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1020) at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:727) at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:693) at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444) at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:961) at org.apache.activemq.transport.tcp.TcpBufferedOutputStream.flush(TcpBufferedOutputStream.java:115) at java.io.DataOutputStream.flush(DataOutputStream.java:123) at org.apache.activemq.transport.tcp.TcpTransport.oneway(TcpTransport.java:194) at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:335) at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:317) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:181) at org.apache.activemq.transport.WireFormatNegotiator.sendWireFormat(WireFormatNegotiator.java:84) at org.apache.activemq.transport.WireFormatNegotiator.start(WireFormatNegotiator.java:74) at org.apache.activemq.transport.TransportFilter.start(TransportFilter.java:64) at org.apache.activemq.broker.TransportConnection.start(TransportConnection.java:1072) at org.apache.activemq.broker.TransportConnector$1$1.run(TransportConnector.java:218) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)} {noformat} Elastic agents may show: {noformat} 2021-03-03 20:40:19,743 ERROR [tunnelserver:26224-1-thread-6] [TunnelAcceptor] Error while accepting tunnel connections. javax.net.ssl.SSLException: Received fatal alert: internal_error at sun.security.ssl.Alert.createSSLException(Alert.java:133) at sun.security.ssl.Alert.createSSLException(Alert.java:117) at sun.security.ssl.TransportContext.fatal(TransportContext.java:311) at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293) at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185) at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149) at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143) at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054) at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394) at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708) at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72) at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791) at java.io.InputStream.read(InputStream.java:101) at com.atlassian.tunnel.tunnel.server.TunnelAcceptor.run(TunnelAcceptor.java:62) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) {noformat} h3. Workaround (i) The below workaround is performed on the Bamboo server side (not agent) h4. Option 1 - Rollback Java Rollback Java JDK8u272 to a previous version of JDK8 h4. Option 2 - Disable specific Java algorithms (!) {*}NOTE{*}: This workaround applies to Bamboo Server and Elastic Agents *Disable RSASSA-PSS:* * Edit the {{$JAVA_HOME/jre/lib/security/java.security}} and add {{RSASSA-PSS}} to the {{jdk.tls.disabledAlgorithms}} *or* create another file containing: {code:java} jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, RSASSA-PSS {code} * Add this to the {{{}$BAMBOO_INSTALL/bin/setenv.sh{}}}, in {{{}JVM_SUPPORT_RECOMMENDED_ARGS{}}}: {code:java} -Djava.security.properties=/path/to/file/created.above {code} This alone will fix the connection between agents and the server via the JMS broker – Bamboo Server only. If you are experiencing SSL error with HTTPS requests directly to Tomcat, {*}in addition to disabling RSASSA-PSS above{*}, you'll also need to disable TLSv1.3 by setting {{{}sslEnabledProtocols="TLSv1.2"{}}}. Example: {noformat} <Connector port="8443" maxThreads="150" minSpareThreads="25" connectionTimeout="20000" disableUploadTimeout="true" acceptCount="100" enableLookups="false" maxHttpHeaderSize="8192" useBodyEncodingForURI="true" URIEncoding="UTF-8" keystoreFile="/path/to/keystore.p12" keystorePass="changeit" scheme="https" secure="true" SSLEnabled="true" sslProtocol="TLS" sslEnabledProtocols="TLSv1.2" clientAuth="false" /> {noformat} |