Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-79477

Confluence and Synchrony logs "Cluster authentication failed. Please make sure all members share the same value" with Hazelcast multicast discovery

XMLWordPrintable

      Issue Summary

      Confluence and Synchrony application logs, continually log the following warnings:

      atlassian-confluence.log
      2022-07-06 09:22:20,785 WARN [hz.confluence.cached.thread-4] [hazelcast.interceptor.authenticator.DefaultClusterJoinManager] checkNodeAuthenticationEnabled CONNECT(192.168.56.30:58883 -> 192.168.56.30:5701): Node authentication failed: Cluster authentication failed. Please make sure all members share the same value for 'confluence.cluster.name' and 'confluence.cluster.authentication.secret' in confluence.cfg.xml. 
2022-07-06 09:22:35,734 WARN [hz.confluence.cached.thread-6] [hazelcast.interceptor.authenticator.DefaultClusterJoinManager] checkNodeAuthenticationEnabled ACCEPT(192.168.56.30:5801 <- 192.168.56.30:38658): Node authentication failed: Cluster authentication failed. Please make sure all members share the same value for 'confluence.cluster.name' and 'confluence.cluster.authentication.secret' in confluence.cfg.xml. 
2022-07-06 09:22:35,734 WARN [hz.confluence.cached.thread-6] [hazelcast.nio.tcp.TcpIpAcceptor] log [192.168.56.30]:5801 [testcluster1] [3.12.11] com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.NodeConnectionException: Cluster authentication failed. Please make sure all members share the same value for 'confluence.cluster.name' and 'confluence.cluster.authentication.secret' in confluence.cfg.xml.
com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.NodeConnectionException: Cluster authentication failed. Please make sure all members share the same value for 'confluence.cluster.name' and 'confluence.cluster.authentication.secret' in confluence.cfg.xml.
	at com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.DefaultClusterJoinManager.checkNodeAuthenticationEnabled(DefaultClusterJoinManager.java:68)
	at com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.DefaultClusterJoinManager.accept(DefaultClusterJoinManager.java:47)
	at com.atlassian.confluence.impl.cluster.hazelcast.interceptor.ClusterJoinSocketInterceptor.onAccept(ClusterJoinSocketInterceptor.java:49)
	at com.hazelcast.nio.NodeIOService.interceptSocket(NodeIOService.java:300)
	at com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread.configureAndAssignSocket(TcpIpAcceptor.java:316)
	at com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread.access$1400(TcpIpAcceptor.java:138)
	at com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread$1.run(TcpIpAcceptor.java:305)
	at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
	at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
	at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
      atlassian-synchrony.log
      2022-07-06 09:22:20,784 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 2022-07-05 23:22:20,783 INFO [hz._hzInstance_1_testcluster1-Synchrony.cached.thread-8] [hazelcast.interceptor.authenticator.DefaultClusterJoinManager] enableNodeAuthentication : true
2022-07-06 09:22:20,784 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 2022-07-05 23:22:20,783 WARN [hz._hzInstance_1_testcluster1-Synchrony.cached.thread-8] [hazelcast.interceptor.authenticator.DefaultClusterJoinManager] ACCEPT(192.168.56.30:5701 <- 192.168.56.30:58883): Node authentication failed: Cluster authentication failed. Please make sure all members share the same value for cluster name and shared secret. 
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 2022-07-05 23:22:20,784 ERROR [hz._hzInstance_1_testcluster1-Synchrony.cached.thread-8] [synchrony.logging] uncaught exception {:throwable #error {
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	 :cause "org/apache/commons/lang3/StringUtils"
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	 :via
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	 [{:type java.lang.NoClassDefFoundError
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	   :message "org/apache/commons/lang3/StringUtils"
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	   :at [com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.NodeConnectionException getMessage "NodeConnectionException.java" 35]}]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	 :trace
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	 [[com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.NodeConnectionException getMessage "NodeConnectionException.java" 35]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread configureAndAssignSocket "TcpIpAcceptor.java" 320]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread access$1400 "TcpIpAcceptor.java" 138]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread$1 run "TcpIpAcceptor.java" 305]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker run "CachedExecutorServiceDelegate.java" 227]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [java.util.concurrent.ThreadPoolExecutor runWorker nil -1]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [java.util.concurrent.ThreadPoolExecutor$Worker run nil -1]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [java.lang.Thread run nil -1]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.util.executor.HazelcastManagedThread executeRun "HazelcastManagedThread.java" 64]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.util.executor.HazelcastManagedThread run "HazelcastManagedThread.java" 80]]}}

      This is reproducible on Data Center: Yes

      Steps to Reproduce

      1. Install Confluence DC as Clustered Type (one node is enough to reproduce issue) on Linux
      2. Choose multicast for the Node discovery
      3. Complete the install
      4. Create, edit and publish a single page
      5. tail the atlassian-confluence.log and atlassian-synchrony.log

      Expected Results

      • No warnings about Hazelcast when the Cluster discovery is "healthy".

      Actual Results

      Wait about 5 - 10 minutes, the below warnings are continually thrown in the atlassian-confluence.log and atlassian-synchrony.log:

      • Specifically, Confluence logs are referencing Synchrony 5701 hazelcast port
      • Synchrony logs are referencing Confluence Synchrony 5801 hazelcast port
      atlassian-confluence.log
      2022-07-06 09:22:20,785 WARN [hz.confluence.cached.thread-4] [hazelcast.interceptor.authenticator.DefaultClusterJoinManager] checkNodeAuthenticationEnabled CONNECT(192.168.56.30:58883 -> 192.168.56.30:5701): Node authentication failed: Cluster authentication failed. Please make sure all members share the same value for 'confluence.cluster.name' and 'confluence.cluster.authentication.secret' in confluence.cfg.xml. 
2022-07-06 09:22:35,734 WARN [hz.confluence.cached.thread-6] [hazelcast.interceptor.authenticator.DefaultClusterJoinManager] checkNodeAuthenticationEnabled ACCEPT(192.168.56.30:5801 <- 192.168.56.30:38658): Node authentication failed: Cluster authentication failed. Please make sure all members share the same value for 'confluence.cluster.name' and 'confluence.cluster.authentication.secret' in confluence.cfg.xml. 
2022-07-06 09:22:35,734 WARN [hz.confluence.cached.thread-6] [hazelcast.nio.tcp.TcpIpAcceptor] log [192.168.56.30]:5801 [testcluster1] [3.12.11] com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.NodeConnectionException: Cluster authentication failed. Please make sure all members share the same value for 'confluence.cluster.name' and 'confluence.cluster.authentication.secret' in confluence.cfg.xml.
com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.NodeConnectionException: Cluster authentication failed. Please make sure all members share the same value for 'confluence.cluster.name' and 'confluence.cluster.authentication.secret' in confluence.cfg.xml.
	at com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.DefaultClusterJoinManager.checkNodeAuthenticationEnabled(DefaultClusterJoinManager.java:68)
	at com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.DefaultClusterJoinManager.accept(DefaultClusterJoinManager.java:47)
	at com.atlassian.confluence.impl.cluster.hazelcast.interceptor.ClusterJoinSocketInterceptor.onAccept(ClusterJoinSocketInterceptor.java:49)
	at com.hazelcast.nio.NodeIOService.interceptSocket(NodeIOService.java:300)
	at com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread.configureAndAssignSocket(TcpIpAcceptor.java:316)
	at com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread.access$1400(TcpIpAcceptor.java:138)
	at com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread$1.run(TcpIpAcceptor.java:305)
	at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
	at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
	at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
      atlassian-synchrony.log
      2022-07-06 09:22:20,784 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 2022-07-05 23:22:20,783 INFO [hz._hzInstance_1_testcluster1-Synchrony.cached.thread-8] [hazelcast.interceptor.authenticator.DefaultClusterJoinManager] enableNodeAuthentication : true
2022-07-06 09:22:20,784 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 2022-07-05 23:22:20,783 WARN [hz._hzInstance_1_testcluster1-Synchrony.cached.thread-8] [hazelcast.interceptor.authenticator.DefaultClusterJoinManager] ACCEPT(192.168.56.30:5701 <- 192.168.56.30:58883): Node authentication failed: Cluster authentication failed. Please make sure all members share the same value for cluster name and shared secret. 
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 2022-07-05 23:22:20,784 ERROR [hz._hzInstance_1_testcluster1-Synchrony.cached.thread-8] [synchrony.logging] uncaught exception {:throwable #error {
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	 :cause "org/apache/commons/lang3/StringUtils"
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	 :via
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	 [{:type java.lang.NoClassDefFoundError
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	   :message "org/apache/commons/lang3/StringUtils"
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	   :at [com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.NodeConnectionException getMessage "NodeConnectionException.java" 35]}]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	 :trace
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	 [[com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.NodeConnectionException getMessage "NodeConnectionException.java" 35]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread configureAndAssignSocket "TcpIpAcceptor.java" 320]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread access$1400 "TcpIpAcceptor.java" 138]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread$1 run "TcpIpAcceptor.java" 305]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker run "CachedExecutorServiceDelegate.java" 227]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [java.util.concurrent.ThreadPoolExecutor runWorker nil -1]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [java.util.concurrent.ThreadPoolExecutor$Worker run nil -1]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [java.lang.Thread run nil -1]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.util.executor.HazelcastManagedThread executeRun "HazelcastManagedThread.java" 64]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.util.executor.HazelcastManagedThread run "HazelcastManagedThread.java" 80]]}}

      Workaround

      1. Shutdown the entire Confluence Cluster
      2. Create/edit the <confluence-shared-home>/synchrony-args.properties file
        • Add in this line: 
          cluster.join.multicast.port=54328
        • This will configure Synchrony hazelcast multicast discovery to work over port 54328 (Confluence hazelcast multicast will still work over the default port 54327)
        • This will now stop Confluence and Synchrony hazelcast discovery conflicting with each other and stop the above mentioned logging lines
      3. Start Confluence (one node at a time)

      Impact

      If the Confluence and Synchrony DC Cluster Nodes are discovering each other and are "healthy", the above warnings can be safely ignored.

      If you are seeing Collaborative Editing issues, refer to:

            [CONFSERVER-79477] Confluence and Synchrony logs "Cluster authentication failed. Please make sure all members share the same value" with Hazelcast multicast discovery

            James Whitehead added a comment -

            A fix for this issue is available in Confluence Server and Data Center 8.0.0.
            Upgrade now or check out the Release Notes to see what other issues are resolved.

            James Whitehead added a comment - A fix for this issue is available in Confluence Server and Data Center 8.0.0. Upgrade now or check out the Release Notes to see what other issues are resolved.

            Felix Freitag added a comment -

            We're experiencing this issue on one of our customer's systems and I'm wondering what's the root cause for this issue (perhaps a configuration mismatch?). It doesn't affect all nodes though. Some of them run perfectly fine and others run into this error every 5-10 mins.

            Felix Freitag added a comment - We're experiencing this issue on one of our customer's systems and I'm wondering what's the root cause for this issue (perhaps a configuration mismatch?). It doesn't affect all nodes though. Some of them run perfectly fine and others run into this error every 5-10 mins.

              5339cdd01cf4 Jeffery Xie
              hlam@atlassian.com Eric Lam
              Affected customers:
              6 This affects my team
              Watchers:
              17 Start watching this issue

                Created:
                Updated:
                Resolved: