Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-9203

Bitbucket Search fails when connecting to Elasticsearch over HTTPS in certain environments

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Low Low
    • 4.14.4
    • 4.9.1
    • None

      Summary

      Bitbucket Search fails to connect to a SSL enabled Elasticsearch instance with the following error in catalina.out:

      Exception in thread "I/O dispatcher 11" java.lang.NoClassDefFoundError: javax/security/auth/x500/X500Principal
      

      Environment

      • Elasticsearch 2.3.1
      • Buckler 0.2.1
      • Java 1.8.0_101
      • Ubuntu 14.04

      Steps to Reproduce

      1. Enable HTTPS on Elasticsearch with a self-signed certificate and import this certificate into the truststore of the JVM running Bitbucket Server
        1. For example, with Buckler:
          tls.http.enabled: true
          tls.tcp.enabled: true
          tls.keystore.path: /path/to/elasticsearch.jks
          tls.keystore.password: changeit
      2. Configure Bitbucket Server to point to the HTTPS URL for Elasticsearch
      3. The test will fail and searches also fail

      Expected Results

      Connection test and search works without error.

      Actual Results

      The following is thrown in the atlassian-bitbucket.log:

      2016-10-03 23:09:15,539 DEBUG [I/O dispatcher 13]  c.a.b.s.s.t.DefaultElasticsearchConnectionTester Detailed Exception: 
      java.util.concurrent.CompletionException: org.apache.http.ConnectionClosedException: Connection closed unexpectedly
              at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292) ~[na:1.8.0_101]
              at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308) ~[na:1.8.0_101]
              at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593) ~[na:1.8.0_101]
              at java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577) ~[na:1.8.0_101]
              at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474) ~[na:1.8.0_101]
              at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977) ~[na:1.8.0_101]
              at com.atlassian.elasticsearch.client.apache.httpclient.ApacheRequestExecutor$1.failed(ApacheRequestExecutor.java:144) ~[na:na]
              at org.apache.http.concurrent.BasicFuture.failed(BasicFuture.java:134) ~[httpcore-4.4.4.jar:4.4.4]
              at org.apache.http.impl.nio.client.AbstractClientExchangeHandler.failed(AbstractClientExchangeHandler.java:414) ~[na:na]
              at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.closed(HttpAsyncRequestExecutor.java:137) ~[na:na]
              at org.apache.http.impl.nio.client.InternalIODispatch.onClosed(InternalIODispatch.java:71) ~[na:na]
              at org.apache.http.impl.nio.client.InternalIODispatch.onClosed(InternalIODispatch.java:39) ~[na:na]
              at org.apache.http.impl.nio.reactor.AbstractIODispatch.disconnected(AbstractIODispatch.java:102) ~[na:na]
              at org.apache.http.impl.nio.reactor.BaseIOReactor.sessionClosed(BaseIOReactor.java:281) ~[na:na]
              at org.apache.http.impl.nio.reactor.AbstractIOReactor.processClosedSessions(AbstractIOReactor.java:442) ~[na:na]
              at org.apache.http.impl.nio.reactor.AbstractIOReactor.hardShutdown(AbstractIOReactor.java:578) ~[na:na]
              at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:307) ~[na:na]
              at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106) ~[na:na]
              at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590) ~[na:na]
              at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_101]
      Caused by: org.apache.http.ConnectionClosedException: Connection closed unexpectedly
              ... 11 common frames omitted
      

      And the corresponding entry in catalina.out:

      Exception in thread "I/O dispatcher 13" java.lang.NoClassDefFoundError: javax/security/auth/x500/X500Principal
              at org.apache.http.conn.ssl.DefaultHostnameVerifier.verify(DefaultHostnameVerifier.java:115)
              at org.apache.http.conn.ssl.DefaultHostnameVerifier.verify(DefaultHostnameVerifier.java:87)
              at org.apache.http.nio.conn.ssl.SSLIOSessionStrategy.verifySession(SSLIOSessionStrategy.java:204)
              at org.apache.http.nio.conn.ssl.SSLIOSessionStrategy$1.verify(SSLIOSessionStrategy.java:188)
              at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:366)
              at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:507)
              at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:122)
              at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:164)
              at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:339)
              at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:317)
              at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:278)
              at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106)
              at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590)
              at java.lang.Thread.run(Thread.java:745)

      Workaround

      Use HTTP to connect to Elasticsearch.

            [BSERV-9203] Bitbucket Search fails when connecting to Elasticsearch over HTTPS in certain environments

            Owen made changes -
            Workflow Original: Stash Workflow - Restricted [ 1608498 ] New: JAC Bug Workflow v3 [ 3135533 ]
            Owen made changes -
            Symptom Severity Original: Minor [ 14432 ] New: Severity 3 - Minor [ 15832 ]
            Paul Thompson (Inactive) made changes -
            Fix Version/s New: 4.14.4 [ 67332 ]
            Resolution New: Fixed [ 1 ]
            Status Original: To be reviewed [ 10026 ] New: Closed [ 6 ]
            Paul Thompson (Inactive) made changes -
            Link Original: This issue is related to BSERVDEV-14665 [ BSERVDEV-14665 ]
            Paul Thompson (Inactive) made changes -
            Link New: This issue is related to BSERVDEV-14665 [ BSERVDEV-14665 ]
            Paul Thompson (Inactive) made changes -
            Status Original: In Progress [ 3 ] New: To be reviewed [ 10026 ]
            Paul Thompson (Inactive) made changes -
            Remote Link New: This issue links to "Bitbucket Search › Main › BSERV-9203-bitbucket-fails-on-certain-certificates (server-syd-bamboo)" [ 272655 ]
            Paul Thompson (Inactive) made changes -
            Status Original: Open [ 1 ] New: In Progress [ 3 ]

            I setup stunnel to work around this bug, otherwise we would have been forced to drop it too.

            Jonas Andersson added a comment - I setup stunnel to work around this bug, otherwise we would have been forced to drop it too.
            Paul Thompson (Inactive) made changes -
            Assignee New: Paul Thompson [ pathompson ]

              pathompson Paul Thompson (Inactive)
              jethomas Jeff Thomas
              Affected customers:
              4 This affects my team
              Watchers:
              10 Start watching this issue

                Created:
                Updated:
                Resolved: