-
Bug
-
Resolution: Fixed
-
Low
-
4.8.4
-
Severity 2 - Major
-
800
-
Issue Summary
Fisheye 4.8.4 bundles new Jetty libraries version 9.4.30, and when a wildcard certificate is used the application fails to start.
Steps to Reproduce
- Make sure to be using Fisheye 4.8.4 configured to use a wildcard certificate
- Try starting the application
Expected Results
Fisheye should be able to read the wildcard certificate.
Actual Results
The application will fail to start, and the below exception is thrown in the atlassian-fisheye.log file when the instance is started in debug mode (by adding the --debug flag):
2020-09-15 09:05:25,293 INFO [main ] org.eclipse.jetty.server.handler.ContextHandler ContextHandler-doStart - Started c.c.f.w.j.FishEyeWebApplicationContext@3daf7722{Fisheye WebApp,/,file:///opt/atlassian/crucible/fecru-4.8.4/content/,AVAILABLE}{/opt/atlassian/crucible/fecru-4.8.4/content}
2020-09-15 09:05:25,300 INFO [main ] org.eclipse.jetty.server.AbstractConnector AbstractConnector-doStart - Started LocalConnector@1ae8bcbc{HTTP/1.1, (http/1.1)}
2020-09-15 09:05:25,347 INFO [main ] org.eclipse.jetty.server.AbstractConnector AbstractConnector-doStart - Started ServerConnector@479ceda0{HTTP/1.1, (http/1.1)}{0.0.0.0:8060}
2020-09-15 09:05:25,349 INFO [main ] org.eclipse.jetty.util.ssl.SslContextFactory SslContextFactory-load - x509=X509@521a8a9b(fisheye,h=[crucible2.colsa.com, colsa.com],w=[colsa.com]) for SslContextFactory@7a5c6d8[provider=null,keyStore=file:///var/atlassian/application/crucible/crucible_new.jks,trustStore=file:///var/atlassian/application/crucible/crucible_new.jks]
2020-09-15 09:05:25,353 DEBUG [main ] fisheye Run-mainImpl - startup stacktrace
java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead)
at org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1274) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1256) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:374) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:97) [jetty-server-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:321) [jetty-server-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81) [jetty-server-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234) [jetty-server-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.server.Server.doStart(Server.java:386) [jetty-server-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at com.cenqua.fisheye.web.WebServer.start(WebServer.java:327) [fisheye.jar:?]
at com.cenqua.fisheye.ctl.Run.mainImpl(Run.java:236) [fisheye.jar:?]
at com.cenqua.fisheye.ctl.Run.main(Run.java:55) [fisheye.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [?:1.8.0_261]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [?:1.8.0_261]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [?:1.8.0_261]
at java.lang.reflect.Method.invoke(Method.java:498) [?:1.8.0_261]
at com.cenqua.fisheye.FishEyeCtl.mainImpl(FishEyeCtl.java:101) [fisheyeboot.jar:?]
at com.cenqua.fisheye.FishEyeCtl.main(FishEyeCtl.java:44) [fisheyeboot.jar:?]
2020-09-15 09:05:25,361 ERROR [main ] fisheye Run-logStartupException - Could not start server: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead)
java.lang.IllegalStateException: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead)
at org.eclipse.jetty.util.ssl.SslContextFactory.newSniX509ExtendedKeyManager(SslContextFactory.java:1274) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.ssl.SslContextFactory.getKeyManagers(SslContextFactory.java:1256) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.ssl.SslContextFactory.load(SslContextFactory.java:374) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.ssl.SslContextFactory.doStart(SslContextFactory.java:245) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.server.SslConnectionFactory.doStart(SslConnectionFactory.java:97) [jetty-server-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.component.ContainerLifeCycle.start(ContainerLifeCycle.java:169) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.component.ContainerLifeCycle.doStart(ContainerLifeCycle.java:117) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.server.AbstractConnector.doStart(AbstractConnector.java:321) [jetty-server-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.server.AbstractNetworkConnector.doStart(AbstractNetworkConnector.java:81) [jetty-server-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.server.ServerConnector.doStart(ServerConnector.java:234) [jetty-server-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.server.Server.doStart(Server.java:386) [jetty-server-9.4.30.v20200611.jar:9.4.30.v20200611]
at org.eclipse.jetty.util.component.AbstractLifeCycle.start(AbstractLifeCycle.java:72) [jetty-util-9.4.30.v20200611.jar:9.4.30.v20200611]
at com.cenqua.fisheye.web.WebServer.start(WebServer.java:327) [fisheye.jar:?]
at com.cenqua.fisheye.ctl.Run.mainImpl(Run.java:236) [fisheye.jar:?]
at com.cenqua.fisheye.ctl.Run.main(Run.java:55) [fisheye.jar:?]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [?:1.8.0_261]
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) [?:1.8.0_261]
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [?:1.8.0_261]
at java.lang.reflect.Method.invoke(Method.java:498) [?:1.8.0_261]
at com.cenqua.fisheye.FishEyeCtl.mainImpl(FishEyeCtl.java:101) [fisheyeboot.jar:?]
at com.cenqua.fisheye.FishEyeCtl.main(FishEyeCtl.java:44) [fisheyeboot.jar:?]
2020-09-15 09:05:25,392 INFO [Thread-1 ] fisheye ShutdownService-stopImpl - Shutdown requested
Workaround
- Use a Fisheye version up to 4.8.3, which uses the old Jetty libraries 9.4.19 and where the problem does not happen, or
- Do not use a wildcard certificate, or
- Switch to plain HTTP
- Terminate SSL at proxy level instead of directly in Fisheye
- mentioned in
-
Page Failed to load
| Form Name | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
[FE-7318] Fisheye fails to start with wildcard certificate
Hi Marek, any update on when the latest version will be released? We are planning to upgrade to latest version of Fisheye to meet below mandates
1. Move to Oracle 19c. Our company has mandate to move to Oracle 19c and FE-7318 is blocking us from upgrade to latest version.
2. Take benefit of security fixes in the newer releases
3. Move to version of Fisheye that is supported by Atlassian
But due to this defect we are not able to proceed. Any insight will be helpful.
Hi Sreenath, please take my sincere apologies for the delay. I estimate we need 1-2 more weeks to release 4.8.7.
Hi Marek, Any update on when the fix to this defect is released. We are blocked in upgrading to latest version and move to Oracle 19c due to this. Thank you.
Yes, a colleague explained the same. I'll look out for 4.8.7, thanks.
This has been fixed in 4.8.7, not 4.8.6. The 4.8.7 has not been released yet as we want to include other fixes.
I've just tried 4.8.6 which claims to have fixed this but I still get
ERROR [main ] fisheye Run-logStartupException - Could not start server: KeyStores with multiple certificates are not supported on the base class org.eclipse.jetty.util.ssl.SslContextFactory. (Use org.eclipse.jetty.util.ssl.SslContextFactory$Server or org.eclipse.jetty.util.ssl.SslContextFactory$Client instead)
@holme118 - While I agree with you on the options 1-3, option 4 was the way to go for us. Installing NGINX and having the traffic terminate with the proxy and then pass http traffic from there, was the best way to go. I was still able to secure my application, while being able to upgrade the latest version of FeCru.
This is a major problem with my company, as we hava a mandate to move to oracle 19c, so we HAVE to run 4.8.5, but we also have been unable to get option 1 or 2 above to work. And option 3 and 4 are not secure enough for us to use. Do we have to drop crucible/fisheye because it is broken and we can't wait?
stuck on version 4.8.3
This is a very solid argument! I will see what we can do to release it earlier.
You're correct Darryl. We're currently stuck on version 4.8.3 until this gets resolved.
Hi Marek,
Thanks for that update. I would ask, if possible, to move this up as soon as possible because, in effect, it may be causing people to not upgrade their instances (let's not assume proxying is an option for everyone) and therefore, the security fixes associated with upgraded versions are not being leveraged because of that. So, not putting this fix in-place, increases risk as time passes.
Cheers, Darryl
Hi Paul, I can't give you an exact date. As you might have noticed, our release cadence for bug-fix releases is about 6-8 weeks. Having said this, we might push it into 4.8.7 to have it earlier, but it's not decided yet.
Hi Marek,
Thank you for the update. Do you have an anticipated release date for 4.8.8?
Paul
Hi Paul,
Indeed, this was originally planned for the 4.8.6 bug-fix release. However, in 4.8.6 my team focused on delivering security fixes. As a consequence, the fix did not fit into the scope. I estimate this bug will be solved in 4.8.8. I'm sorry for the inconvenience.
Kind regards
Marek Parfianowicz
Marek,
We are affected by this bug and were anticipating the fix in the February release. Since it was not listed in the release notes, I assume it was not included in the 4.8.6 release. Do you know if it's slated for a future release?
Thank you!
Paul
Hi Marek,
Sorry for the delay...
We have multiple SANs in a single cert (I use the same certificate for jira, confluence, bitbucket, etc...) so they all appear as: jira.my.domain.com, confluence.my.domain.com, fisheye.my.domain.com, etc...
| The only case when using http is fine is when you put a proxy in front of Fisheye, redirecting from http to https. |
To a point that's true. Technically, the proxy should reside on the same host as the application, otherwise, you are potentially sending network credentials, etc.. across your internal network in the clear. This is not always the case however.
Cheers,
Darryl
Hi Darryl,
Can you confirm please whether 4.8.3, which I was able to upgrade to, is affected by this critical issue [...]
The CVE-2017-12611 is tracked in FE-7331 and CRUC-8497 issues. The problem has been fixed in version 4.8.4 of Fisheye/Crucible, so yes, older versions are potentially vulnerable. Please note that according to our assessment its CVSS v3 score is 8.1, which means high severity.
we do NOT have a wildcard cert (we have a cert with SANs)
Can you confirm that having two or more domains (without a wildcard) in the same certificate also causes Fisheye 4.8.4 fail to start? Did I understood you correctly?
switching to http violates our policies and goes against the spirit of https everywhere
I fully agree. The only case when using http is fine is when you put a proxy in front of Fisheye, redirecting from http to https.
Kind regards
Marek Parfianowicz
Hi Marek,
Thanks for the response. Can you confirm please whether 4.8.3, which I was able to upgrade to, is affected by this critical issue that is apparently resolved in 4.8.4 (which I assume would require a release sooner than later):
My concern here is that we run the system using HTTPS, we do NOT have a wildcard cert (we have a cert with SANs), switching to http violates our policies and goes against the spirit of https everywhere. So the issue of the jetty update in fecru is essentially blocking us from applying a security fix. Forcing the customer to build out a proxy as a workaround, isn't overly customer-centric.
Hi Darryl, this bug is a candidate for 4.8.6 bug-fix release, expected in February 2021.
Kind regards
Marek Parfianowicz
Development Team Lead
Hi there,
Do we have a timeframe for this fix? I just hit this while doing an upgrade and didn't see any warnings or documentation regarding this issue in the release notes. This is most definitely not a "low" in my opinion.
My team and I were unsuccessful with the alternate certificate methods. No wildcard certificates were used either. Eventually we realized this wasn't an actual certificate error, but with how Jetty Web Server wasn't able to view the keystore correctly. In order to get the application back online with HTTPS enabled, we commented out the HTTPS bind/settings in the $FISHEYE_INST/config.xml }}file and decided to open up the HTTP, then installing NGINX as the web proxy. After a few configurations, we were successful with allowing HTTPS traffic with NGINX. Just make sure when setting {{proxy_pass setting, you're setting it to:{{}}
proxy_pass http://localhost:<port_number>
It looks like this is, indeed, related to an update in Jetty - and actually has impacted similar applications such as Jenkins, where it looks like they ultimately ended up needing to change to using the Server variant of the "SslContextFactory" implementation in order to resolve this error.
This is due to the change made here in the code for Jetty. Checking the changes we made in 4.8.4, it looks like we bumped up Jetty to a version that includes this change (see code change here):
Old:
<JETTY_VERSION>9.4.19.v20190610</JETTY_VERSION>
New:
<JETTY_VERSION>9.4.30.v20200611</JETTY_VERSION>
With the above being said, our options seem limited to either rolling back to an older version of Jetty, or making the needed code changes to ensure that we're calling the Server flavor of SSLContextFactory on this line in our WebServer.java, similar to what Jenkins did:
//old and busted: SslContextFactory ssl = new SslContextFactory(); //new hotness: SslContextFactory.Server ssl = new SslContextFactory.Server();
Hello!
I would like to apologize everyone watching this bug and waiting for a release. Please let me explain the delay. In the incoming version 4.8.7 we upgraded numerous third party libraries, including an upgrade of the Google Web Toolkit library to the latest version. Unfortunately, this version turned out to be less stable and we're working on a fix. This is the only issue preventing us from releasing 4.8.7. We want to give you thoroughly tested product of high quality.
My current estimation is that we will release 4.8.7 till end of June. I am sorry for inconvenience.
Kind regards
Marek Parfianowicz
Development Team Lead