-
Type:
Bug
-
Resolution: Cannot Reproduce
-
Priority:
Low
-
None
-
Affects Version/s: 4.14.9
-
Component/s: None
-
None
-
1
-
Severity 2 - Major
-
15
Summary
Webhooks from the Primary Bitbucket instance to a mirror are failing in their SSL handshake. The bundled version of the Apache HttpComponents libraries used by the primary server webhooks to the mirror do not handle SSL handshake failures properly resulting in a memory leak.
Heap dump includes the exception being thrown, buried down inside the guts of the Apache HttpComponents code. The exception looks like this:
javax.net.ssl.SSLHandshakeException: General SSLEngine problem caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: timestamp check failed caused by: java.security.cert.CertPathValidatorException: timestamp check failed caused by: java.security.cert.CertificateExpiredException: NotAfter: Tue Jun 19 xx:x:xx GMT 2018
The cluster considers the SSL certificate installed on the mirror to be expired and that's why the handshake is failing.
Steps to Reproduce
- Connect Bitbucket to a mirror.
- Add and expired SSL certificate to a mirror.
Expected Results
Bitbucket will stop trying to connect to the mirror and Apache HttpComponents libraries will handle the SSL handshake failure properly.
Actual Results
Thousands of HTTP requests will be seen going to the mirror resulting in a memory leak.
Notes
Several requests like the below will be seen in the bitbucket-access.logs per second (2-4 a second).
11.34.12.000,171.41.459.550 | https | o@KFGIE2x763x1830257x0 | upstream_f8b69106-0cb4-39d1-a459-844d1325371d | 2018-05-29 12:43:30,804 | "POST /rest/mirroring/latest/upstreamServers/f8b29106-0cb4-39d1-a459-844d1324371d/addon/events HTTP/1.1" | "" "Atlassian HttpClient 0.23.0 / Bitbucket-4.14.9 (4014009) / Default" | 204 | 1349 | 0 | - | 4 | gsj224 |
Thread dumps and heap dumps are necessary to properly diagnose this issue.
Many org.apache.http.impl.nio.reactor.BaseIOReactor objects in the heap as well as many threads like this in the thread dumps:
I/O dispatcher 150" #6107 prio=5 os_prio=0 tid=0x00007fb800166800 nid=0x7a8 runnable [0x00007fb7d7e84000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) - locked <0x00000000a08ac470> (a sun.nio.ch.Util$3) - locked <0x00000000a08ac460> (a java.util.Collections$UnmodifiableSet) - locked <0x00000000a08ac480> (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:257) at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:106) at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:590) at java.lang.Thread.run(Thread.java:745)
Workaround
- Verify the certificate installed on mirrors and or renew certificate.
- Remove the mirror.
- Disable mirroring. In order to disable the mirror:
- On your primary server, go to Administration >> Manage Add-ons
- Search for your mirror name or the word 'mirror'. The add-on should appear as Mirror - <mirror name>
- Expand the add-on panel for the mirror you wish to disable and click the Disable button
Solution
Upgrade to later versions of Bitbucket where bug in the libraries has been fixed.
