Crowd HTTP client ignores SSL system properties

Summary

When Crowd is configured to use two-way SSL by setting clientAuth="true" in server.xml, Embedded Crowd clients fail to connect to it with a bad_certificate error.

Environment

• Crowd 2.8.3 with clientAuth="true" in server.
• FishEye 4.2.0
• JIRA 6.4.3

Steps to Reproduce

1. Generate a self-signed certificate for Crowd
2. Configure Crowd for SSL and set clientAuth to true

   <Connector port="8443" protocol="org.apache.coyote.http11.Http11Protocol"
              maxThreads="150" SSLEnabled="true" scheme="https" secure="true"
              clientAuth="true" sslProtocol="TLS" truststoreFile="/usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts" />

3. Generate a self-signed certificate for FishEye/Crucible
4. Configure FishEye for SSL

   <web-server context="/fecru" site-url="https://jeffjeff.office.atlassian.com/fecru">
   <http bind=":8060" /><ssl keystore="/var/atlassian/application-data/fecru/ssl/keystore.kst" bind=":8061" keystore-password="changeit" truststore-password="changeit" truststore="/usr/lib/jvm/java-8-oracle/jre/lib/security/cacerts"><proxy-info/><excludeProtocols><protocol>SSLv3</protocol></excludeProtocols></ssl></web-server>

5. Set FISHEYE_OPTS to specify the keystore

   FISHEYE_OPTS="-Djavax.net.ssl.keyStore=/var/atlassian/application-data/fecru/ssl/keystore.kst -Djavax.net.ssl.keyStorePassword=changeit"

6. Import the FishEye certificate into the truststore of the JVM for Crowd
7. Import the Crowd certificate into the truststore of the JVM for FishEye
8. Restart both Crowd and FishEye
9. Create a User Directory in FishEye pointed to the secure URL for crowd and click Test

Expected Results

The test connection is successful

Actual Results

The UI displays the following message:

Connection test failed. Response from the server:
javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate

The below exception is thrown in the atlassian-fisheye-YYYY-MM-DD.log file:

2016-10-07 13:55:57,100 ERROR [qtp106999035-231 ] com.atlassian.crowd.embedded.admin.ConfigurationController ConfigurationController-handleSubmit - Configuration test failed for user directory: [ Crowd Server], type: [ CROWD ]
com.atlassian.crowd.exception.runtime.OperationFailedException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
        at com.atlassian.crowd.embedded.core.CrowdDirectoryServiceImpl.testConnection(CrowdDirectoryServiceImpl.java:78) [embedded-crowd-core-2.8.8.jar:?]
        at com.atlassian.fecru.user.crowd.DelegatingCrowdDirectoryService.testConnection(DelegatingCrowdDirectoryService.java:29) [fisheye.jar:?]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [?:1.8.0_101]
...
Caused by: com.atlassian.crowd.exception.OperationFailedException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
        at com.atlassian.crowd.integration.rest.service.RestExecutor$MethodExecutor.andReceive(RestExecutor.java:493) [crowd-integration-client-rest-2.8.8.jar:?
]
        at com.atlassian.crowd.integration.rest.service.RestCrowdClient.searchUs
ers(RestCrowdClient.java:586) [crowd-integration-client-rest-2.8.8.jar:?]
        at com.atlassian.crowd.integration.rest.service.RestCrowdClient.testConnection(RestCrowdClient.java:574) [crowd-integration-client-rest-2.8.8.jar:?]
        at com.atlassian.crowd.directory.RemoteCrowdDirectory.testConnection(RemoteCrowdDirectory.java:839) [crowd-remote-2.8.8.jar:?]
        at com.atlassian.crowd.embedded.core.CrowdDirectoryServiceImpl.testConnection(CrowdDirectoryServiceImpl.java:69) [embedded-crowd-core-2.8.8.jar:?]
        ... 182 more
...
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) [?:1.8.0_101]
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:154) [?:1.8.0_101]
        at sun.security.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:2023) [?:1.8.0_101]
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1125) [?:1.8.0_101]
        at sun.security.ssl.SSLSocketImpl.waitForClose(SSLSocketImpl.java:1769) [?:1.8.0_101]
        at sun.security.ssl.HandshakeOutStream.flush(HandshakeOutStream.java:124) [?:1.8.0_101]
        at sun.security.ssl.Handshaker.sendChangeCipherSpec(Handshaker.java:1083) [?:1.8.0_101]
        at sun.security.ssl.ClientHandshaker.sendChangeCipherAndFinish(ClientHandshaker.java:1222) [?:1.8.0_101]

Notes

• The JVM is able to communicate successfully between the Crowd and FishEye server.
  • Crowd to FishEye

    $ java SSLPoke fecru.internal.jeffjeff.local 8061
     
    Successfully connected

  • FishEye to Crowd

    $ java -Djavax.net.ssl.keyStore=/var/atlassian/application-data/fecru/ssl/keystore.kst -Djavax.net.ssl.keyStorePassword=changeit SSLPoke crowd.internal.jeffjeff.local 8443
     
    Successfully connected

• An Application Link can successfully be establish between FishEye and Crowd, suggesting that this bug lies in the REST Client of Embedded Crowd
• FishEye can be replaced with JIRA, Confluence, or Bitbucket Server and the results would be the same
• With JVM SSL debugging turned on, the failure is caused when the client certificate is not found

  *** ServerHelloDone
  Warning: no suitable certificate found - continuing without client authentication
  *** Certificate chain
  <Empty>

• On a successful connection (SSLPoke or AppLink), the client certificate is found

  *** Finished
  verify_data:  { 29, 223, 122, 244, 160, 159, 178, 222, 47, 13, 84, 173 }
  ***
  qtp106999035-239, WRITE: TLSv1.2 Change Cipher Spec, length = 1
  matching alias: fisheye
  *** Certificate chain
  chain [0] = [
  [
    Version: V3
    Subject: CN=fecru.internal.jeffjeff.local, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown
    Signature Algorithm: SHA256withRSA, OID = 1.2.840.113549.1.1.11
   
    Key:  Sun RSA public key, 2048 bits
    modulus: 28009499656246212081304414805878170785055127684071061541433755109071710745267765886718931723309765215656120397721803926162186932840132869041866188085176276626758018020892747360386411972741350489814211909118815117332095367172314232001841763295328408835096567282606715683336397623078504902332106363652429744957000420485378771514523587803608041889859573809115668463020057472431610100690447399236198650572408920927185290453800135325546061871637217487300674944924234530135571150800285050549013472994356801104766240614479460520914755318246688575356155093747788840022121596823283237384475404216210794913588838190375781803489
    public exponent: 65537
    Validity: [From: Thu Oct 06 21:13:02 UTC 2016,
                 To: Wed Jan 04 21:13:02 UTC 2017]
    Issuer: CN=fecru.internal.jeffjeff.local, OU=Unknown, O=Unknown, L=Unknown, ST=Unknown, C=Unknown
    SerialNumber: [    4b0913a4]

Workaround

• Set clientAuth="false" in Crowd
• Set up an additional connector in Crowd which does not have clientAuth enabled