Details
-
Bug
-
Resolution: Fixed
-
Low
-
None
-
8.8.0, 8.8.1
-
None
-
2
-
Severity 3 - Minor
-
5
-
Description
Issue Summary
This is reproducible on Data Center: yes
As part of Confluence 8.9 release notes, it is mentioned that this release does not support rolling upgrade, however the Plan for upgrade still shows that 8.9.0 is upgrade-compatible
Hazelcast updated
For: ADMINS
This release contains an upgrade to Hazelcast IMDG 5.3.6 as part of our focus on keeping Confluence Data Center secure and up to date. This recent upgrade enhances our platform by improving JMX metrics, which translates to more efficient garbage collector operations, and optimized memory consumption. This version does not support rolling upgrades (ZDU).
Steps to Reproduce
- Create a two-node cluster
- Under plan your upgrade, you can see a rolling upgrade possible to 8.9.0
- Go to Administration
> General Configuration > Rolling upgrades. - Select the Upgrade mode toggle
- Access the node through a command line or SSH.
- Start by shutting down Confluence gracefully on the node
- Download the new installer or zip file for 8.9.0 upgrade
- Start the upgrade on Node 1
- Start Confluence, and confirm that you can log in and view pages before continuing to the next
Expected Results
As soon as the first node is upgraded it joins the cluster, cluster status transitions to Mixed.
Actual Results
Node 1 does not start
The below exception is thrown in the catalina.out file:
2024-04-11 05:36:00,874 INFO [Catalina-utility-1] [confluence.cluster.hazelcast.HazelcastClusterManager] configure Configuring Hazelcast with instanceName [confluence], join configuration TCP/IP member addresses: 10.229.152.40|10.229.152.109, ne
twork interfaces [10.229.152.40, fe80:0:0:0:8f4:d9ff:fefc:c03d%eth0] and local port 5801
2024-04-11 05:36:00,877 INFO [Catalina-utility-1] [confluence.cluster.hazelcast.HazelcastClusterManager] startCluster Starting the cluster.
2024-04-11 05:36:00,883 WARN [Catalina-utility-1] [hazelcast.instance.impl.HazelcastInstanceFactory] log WARNING: Classpath misconfiguration: found multiple META-INF/services/com.hazelcast.instance.impl.NodeExtension resources: jar:file:/opt/atl
assian/confluence/atlassian-confluence-8.8.0/confluence/WEB-INF/lib/com.atlassian.confluence.cache_confluence-cache-hazelcast-8.9.0.jar!/META-INF/services/com.hazelcast.instance.impl.NodeExtension, jar:file:/opt/atlassian/confluence/atlassian-co
nfluence-8.8.0/confluence/WEB-INF/lib/com.hazelcast_hazelcast-5.3.6.jar!/META-INF/services/com.hazelcast.instance.impl.NodeExtension
2024-04-11 05:36:01,373 WARN [Catalina-utility-1] [com.hazelcast.cp.CPSubsystem] log [10.229.152.40]:5801 [InstEnv] [5.3.6] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Ple
ase note that UNSAFE mode will not provide strong consistency guarantees.
2024-04-11 05:36:01,601 INFO [hz.confluence.event-1] [confluence.cluster.hazelcast.LoggingClusterMembershipListener] memberAdded [10.229.152.40]:5801 joined the cluster
2024-04-11 05:36:01,602 INFO [hz.confluence.event-1] [confluence.cluster.hazelcast.LoggingClusterMembershipListener] logClusterMembers Cluster now has 1 members: [[10.229.152.40]:5801]
2024-04-11 05:37:04,114 ERROR [hz.confluence.IO.thread-in-1] [hazelcast.spi.impl.NodeEngine] log [10.229.152.40]:5801 [InstEnv] [5.3.6] Header flags [100001] specify an undefined packet type UNDEFINED5
2024-04-11 05:37:04,117 ERROR [hz.confluence.IO.thread-in-1] [hazelcast.spi.impl.NodeEngine] log [10.229.152.40]:5801 [InstEnv] [5.3.6] Failed to process: Packet{partitionId=-1, frameLength=75, conn=Connection[id=2, /10.229.152.40:5801->/10.229.152.109:34655, qualifier=null, endpoint=null, remoteUuid=null, alive=true, connectionType=MEMBER, planeIndex=-1], rawFlags=100000, isUrgent=false, packetType=SERVER_CONTROL, typeSpecificFlags=<NONE>}
com.hazelcast.nio.serialization.HazelcastSerializationException: There is no suitable de-serializer for type 1485060. This exception is likely caused by differences in the serialization configuration between members or between clients and members.
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.newHazelcastSerializationException(AbstractSerializationService.java:322)
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.readObject(AbstractSerializationService.java:353)
at com.hazelcast.internal.serialization.InternalSerializationService.readObject(InternalSerializationService.java:81)
at com.hazelcast.internal.serialization.impl.ByteArrayObjectDataInput.readObject(ByteArrayObjectDataInput.java:600)
at com.hazelcast.internal.cluster.impl.MemberHandshake.readData(MemberHandshake.java:162)
at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.readInternal(DataSerializableSerializer.java:167)
at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:113)
at com.hazelcast.internal.serialization.impl.DataSerializableSerializer.read(DataSerializableSerializer.java:52)
at com.hazelcast.internal.serialization.impl.StreamSerializerAdapter.read(StreamSerializerAdapter.java:44)
at com.hazelcast.internal.serialization.impl.AbstractSerializationService.toObject(AbstractSerializationService.java:271)
at com.hazelcast.internal.server.tcp.TcpServerControl.process(TcpServerControl.java:64)
at com.hazelcast.internal.server.tcp.TcpServerConnectionManager.accept(TcpServerConnectionManager.java:121)
at com.hazelcast.internal.server.tcp.TcpServerConnectionManager.accept(TcpServerConnectionManager.java:81)
at com.hazelcast.spi.impl.PacketDispatcher.accept(PacketDispatcher.java:79)
at com.hazelcast.spi.impl.PacketDispatcher.accept(PacketDispatcher.java:36)
at com.hazelcast.internal.server.tcp.PacketDecoder.onPacketComplete(PacketDecoder.java:85)
at com.hazelcast.internal.server.tcp.PacketDecoder.onRead(PacketDecoder.java:67)
at com.hazelcast.internal.networking.nio.NioInboundPipeline.process(NioInboundPipeline.java:137)
at com.hazelcast.internal.networking.nio.NioThread.processSelectionKey(NioThread.java:383)
at com.hazelcast.internal.networking.nio.NioThread.processSelectionKeys(NioThread.java:368)
at com.hazelcast.internal.networking.nio.NioThread.selectLoop(NioThread.java:294)
at com.hazelcast.internal.networking.nio.NioThread.executeRun(NioThread.java:249)
at com.hazelcast.internal.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:111)
2024-04-11 08:15:30,166 ERROR [Catalina-utility-1] [hazelcast.instance.impl.Node] log [10.229.152.40]:5801 [InstEnv] [5.3.6] Could not join cluster. Shutting down now!
2024-04-11 08:15:30,170 WARN [Catalina-utility-1] [hazelcast.instance.impl.Node] log [10.229.152.40]:5801 [InstEnv] [5.3.6] Terminating forcefully...
2024-04-11 08:15:30,196 WARN [Catalina-utility-1] [internal.util.phonehome.PhoneHome] log [10.229.152.40]:5801 [InstEnv] [5.3.6] Could not schedule phone home task! Most probably Hazelcast failed to start.
2024-04-11 08:15:30,200 ERROR [Catalina-utility-1] [atlassian.confluence.setup.ConfluenceConfigurationListener] contextInitialized An error was encountered while bootstrapping Confluence (see below): \nNode failed to start!
java.lang.IllegalStateException: Node failed to start!
at com.hazelcast.instance.impl.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:126)
at com.hazelcast.instance.impl.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:217)
at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:196)
at com.hazelcast.instance.impl.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:134)
at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:61)
at com.atlassian.confluence.cluster.hazelcast.HazelcastClusterManager.createHazelcastInstance(HazelcastClusterManager.java:383)
at com.atlassian.confluence.cluster.hazelcast.HazelcastClusterManager.startCluster(HazelcastClusterManager.java:370)
at com.atlassian.confluence.cluster.hazelcast.HazelcastClusterManager.reconfigure(HazelcastClusterManager.java:346)
at com.atlassian.confluence.cluster.DefaultClusterConfigurationHelper.bootstrapCluster(DefaultClusterConfigurationHelper.java:417)
at com.atlassian.confluence.setup.DefaultBootstrapManager.bootstrapCluster(DefaultBootstrapManager.java:1029)
at com.atlassian.confluence.setup.DefaultBootstrapManager.afterConfigurationLoaded(DefaultBootstrapManager.java:1008)
at com.atlassian.config.bootstrap.DefaultAtlassianBootstrapManager.init(DefaultAtlassianBootstrapManager.java:72)
at com.atlassian.confluence.setup.DefaultBootstrapManager.init(DefaultBootstrapManager.java:261)
at com.atlassian.config.util.BootstrapUtils.init(BootstrapUtils.java:34)
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.initialiseBootstrapContext(ConfluenceConfigurationListener.java:151)
at com.atlassian.confluence.setup.ConfluenceConfigurationListener.contextInitialized(ConfluenceConfigurationListener.java:72)
at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:4460)
at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:4914)
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:171)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1332)
at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1322)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
at java.base/java.lang.Thread.run(Thread.java:829)
