Issue Summary

After selecting Upgrade mode on the rolling upgrade page, shutting down one cluster node, and starting a node with a higher version, the Rolling upgrade and Login pages fail to render.

This is reproducible on Data Center: yes

Steps to Reproduce

• Install a Bitbucket cluster with two nodes running version 9.4.1.
• Enable upgrade mode from the Rolling upgrade page.
• Shut down one of the Bitbucket nodes.
• Start the Bitbucket application with version 9.4.3.
• Attempt to load the Rolling upgrade page.

Expected Results

The Rolling upgrade page and the login page should display correctly.

Actual Results

• Loading the Rolling upgrade page results in an error.

  2025-02-23 08:58:30,521 DEBUG [http-nio-7990-exec-7] aman @1RI0P2Zx538x586x2 1n2r0bn 10.211.55.2,10.211.55.6 "GET /rest/zdu/cluster HTTP/1.1" c.a.s.i.c.HazelcastClusterService Querying all nodes for status
  2025-02-23 08:58:30,522 ERROR [http-nio-7990-exec-7] aman @1RI0P2Zx538x586x2 1n2r0bn 10.211.55.2,10.211.55.6 "GET /rest/zdu/cluster HTTP/1.1" c.a.b.i.r.e.UnhandledExceptionMapper Unhandled exception while processing REST request: "GET /rest/zdu/cluster HTTP/1.1"
  java.util.concurrent.RejectedExecutionException: ExecutorService[bitbucket.core] is shutdown! In order to create a new ExecutorService with name 'bitbucket.core', you need to destroy current ExecutorService first!
     at com.hazelcast.executor.impl.ExecutorServiceProxy.checkNotShutdown(ExecutorServiceProxy.java:235)
     at com.hazelcast.executor.impl.ExecutorServiceProxy.submitToMember(ExecutorServiceProxy.java:407)
     at com.hazelcast.executor.impl.ExecutorServiceProxy.submitToMembers(ExecutorServiceProxy.java:455)
     at com.hazelcast.executor.impl.ExecutorServiceProxy.submitToAllMembers(ExecutorServiceProxy.java:463)
     at com.atlassian.stash.internal.cluster.HazelcastClusterService.getNodeStates(HazelcastClusterService.java:80)
     at jdk.internal.reflect.GeneratedMethodAccessor982.invoke(Unknown Source)
     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at jdk.internal.reflect.GeneratedMethodAccessor307.invoke(Unknown Source)
     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at jdk.proxy3/jdk.proxy3.$Proxy85.getNodeStates(Unknown Source)
     at jdk.internal.reflect.GeneratedMethodAccessor982.invoke(Unknown Source)
     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
     at jdk.proxy2/jdk.proxy2.$Proxy624.getNodeStates(Unknown Source)
     at jdk.internal.reflect.GeneratedMethodAccessor982.invoke(Unknown Source)
     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
     at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
     at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
     at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
     at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
     at jdk.proxy11/jdk.proxy11.$Proxy852.getNodeStates(Unknown Source)
     at com.atlassian.zdu.bitbucket.impl.BitbucketClusterManagerAdapter.getNodes(BitbucketClusterManagerAdapter.java:27)
     at com.atlassian.zdu.NodeInfoAccessor.getNodes(NodeInfoAccessor.java:26)
     at com.atlassian.zdu.impl.ZduServiceImpl.getNodes(ZduServiceImpl.java:70)
     at com.atlassian.zdu.impl.ZduServiceImpl.getCluster(ZduServiceImpl.java:137)
     at com.atlassian.zdu.rest.ZduResource.getCluster(ZduResource.java:84)
     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
     at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
     at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
     at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:146)
     at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:189)
     at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$TypeOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:219)
     at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:93)
     at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:478)
     at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:400)
     at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:81)
     at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:256)
     at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
     at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
     at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
     at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
     at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
     at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
     at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:235)
     at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:684)
     at org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:394)
     at org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:346)
     at org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:359)
     at org.glassfish.jersey.servlet.ServletContainer.doFilter(ServletContainer.java:432)
     at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:28)
     at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:28)
     at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:28)
     at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:28)
     at com.atlassian.applinks.core.rest.context.ContextFilter.doFilter(ContextFilter.java:28)
     at com.atlassian.analytics.client.filter.UniversalAnalyticsFilter.doFilter(UniversalAnalyticsFilter.java:77)
     at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:33)
     at com.atlassian.bitbucket.internal.xcode.web.XcodeUserAgentFilter.doFilter(XcodeUserAgentFilter.java:38)
     at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
     at com.atlassian.bitbucket.internal.ratelimit.servlet.filter.RateLimitFilter.doFilter(RateLimitFilter.java:75)
     at com.atlassian.theme.filter.DefaultRequestOverrideServletFilter.doFilter(DefaultRequestOverrideServletFilter.java:72)
     at com.atlassian.troubleshooting.thready.filter.AbstractThreadNamingFilter.doFilter(AbstractThreadNamingFilter.java:46)
     at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
     at com.atlassian.stash.internal.web.auth.AuthorizationFailureInterceptor.doFilterInternal(AuthorizationFailureInterceptor.java:39)
     at com.atlassian.stash.internal.spring.security.StashAuthenticationFilter.doFilter(StashAuthenticationFilter.java:86)
     at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doInsideSpringSecurityChain(BeforeLoginPluginAuthenticationFilter.java:112)
     at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:75)
     at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:94)
     at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:69)
     at com.atlassian.oauth2.provider.core.web.AccessTokenFilter.doFilter(AccessTokenFilter.java:88)
     at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
     at com.atlassian.plugins.authentication.sso.web.filter.loginform.DisableNativeLoginAuthFilter.doFilterInternal(DisableNativeLoginAuthFilter.java:73)
     at com.atlassian.plugins.authentication.sso.web.filter.AbstractJohnsonAwareFilter.doFilter(AbstractJohnsonAwareFilter.java:29)
     at com.atlassian.plugins.authentication.basicauth.filter.DisableBasicAuthFilter.doFilter(DisableBasicAuthFilter.java:79)
     at com.atlassian.jwt.internal.servlet.JwtAuthFilter.doFilter(JwtAuthFilter.java:39)
     at com.atlassian.analytics.client.filter.DefaultAnalyticsFilter.doFilter(DefaultAnalyticsFilter.java:28)
     at com.atlassian.analytics.client.filter.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:33)
     at com.atlassian.troubleshooting.thready.filter.AbstractThreadNamingFilter.doFilter(AbstractThreadNamingFilter.java:46)
     at com.atlassian.stash.internal.spring.lifecycle.LifecycleJohnsonServletFilterModuleContainerFilter.doFilter(LifecycleJohnsonServletFilterModuleContainerFilter.java:42)
     at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doBeforeBeforeLoginFilters(BeforeLoginPluginAuthenticationFilter.java:90)
     at com.atlassian.stash.internal.web.auth.BeforeLoginPluginAuthenticationFilter.doFilter(BeforeLoginPluginAuthenticationFilter.java:73)
     at com.atlassian.stash.internal.request.DefaultRequestManager.doAsRequest(DefaultRequestManager.java:85)
     at com.atlassian.stash.internal.hazelcast.ConfigurableWebFilter.doFilter(ConfigurableWebFilter.java:38)
     at java.base/java.lang.Thread.run(Thread.java:840)
     ... 257 frames trimmed
  

• The username/password fields do not appear during the Rolling upgrade on the login page.

  2025-02-23 09:38:50,474 DEBUG [http-nio-7990-exec-7] *FBOAM4x578x1435x0 10.211.XX.XX,10.211.XX.XX "GET /dashboard HTTP/1.1" c.a.s.i.i18n.PluginI18nService No values found in any valid locale for key ProviderManager.providerNotFound and locales [en_US, en]
  2025-02-23 09:38:50,480 DEBUG [http-nio-7990-exec-7] *FBOAM4x578x1435x0 10.211.XX.XX,10.211.XX.XX "GET /dashboard HTTP/1.1" c.a.s.i.i18n.PluginI18nService No values found in any valid locale for key AbstractAccessDecisionManager.accessDenied and locales [en_US, en]
  2025-02-23 09:38:50,480 DEBUG [http-nio-7990-exec-7] *FBOAM4x578x1435x0 10.211.XX.XX,10.211.XX.XX "GET /dashboard HTTP/1.1" c.a.s.i.i18n.PluginI18nService No values found in any valid locale for key ExceptionTranslationFilter.insufficientAuthentication and locales [en_US, en]
  2025-02-23 09:38:50,482 DEBUG [http-nio-7990-exec-7] *FBOAM4x578x1435x0 10.211.XX.XX,10.211.XX.XX "GET /dashboard HTTP/1.1" c.a.s.i.s.s.StashLoginUrlAuthenticationEntryPoint Redirecting /mvc/dashboard to /login?next=%2Fdashboard
  2025-02-23 09:38:50,681 DEBUG [http-nio-7990-exec-4] *FBOAM4x578x1439x1 10.211.XX.XX,10.211.XX.XX "GET /mvc/error404 HTTP/1.1" c.a.s.i.i18n.PluginI18nService No values found in any valid locale for key ProviderManager.providerNotFound and locales [en_US, en]
  2025-02-23 09:38:50,692 DEBUG [http-nio-7990-exec-4] *FBOAM4x578x1439x1 10.211.XX.XX,10.211.XX.XX "GET /mvc/error404 HTTP/1.1" c.a.s.i.web.ErrorPageController Page not found attempting to access [/s/4dee1ed1bfb8f6165d223c1338716c74-CDN/-1188018231/40a434c/n1cn5w/82d2e2df3aadba0f3a935d153373d206/_/download/contextbatch/css/bitbucket.layout.base,atl.general,bitbucket.internal.feature.theme,bitbucket.internal.feature.whats.new,-_super/batch.css]
  2025-02-23 09:38:52,144 DEBUG [http-nio-7990-exec-7] *FBOAM4x578x1445x0 10.211.XX.XX,10.211.XX.XX "GET /rest/shortcuts/latest/shortcuts/40a434c/bundled HTTP/1.1" c.a.s.i.i18n.PluginI18nService No values found in any valid locale for key ProviderManager.providerNotFound and locales [en_US, en]
  2025-02-23 09:38:52,149 DEBUG [http-nio-7990-exec-2] *FBOAM4x578x1446x1 10.211.XX.XX,10.211.XX.XX "POST /rest/wrm/2.0/resources HTTP/1.1" c.a.s.i.i18n.PluginI18nService No values found in any valid locale for key ProviderManager.providerNotFound and locales [en_US, en]
  2025-02-23 09:38:53,862 DEBUG [http-nio-7990-exec-3] *FBOAM4x578x1453x0 10.211.XX.XX,10.211.XX.XX "POST /rest/analytics/1.0/publish/bulk HTTP/1.1" c.a.s.i.i18n.PluginI18nService No values found in any valid locale for key ProviderManager.providerNotFound and locales [en_US, en]

  

Workaround

Upgrade the nodes to the same version. The upgraded nodes should be able to rejoin the cluster.

However, to fix the error in the Rolling upgrades page, a full shutdown and restart of the instance is needed.