Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-79477

Confluence and Synchrony logs "Cluster authentication failed. Please make sure all members share the same value" with Hazelcast multicast discovery

    XMLWordPrintable

Details

    Description

      Issue Summary

      Confluence and Synchrony application logs, continually log the following warnings:

      atlassian-confluence.log
      2022-07-06 09:22:20,785 WARN [hz.confluence.cached.thread-4] [hazelcast.interceptor.authenticator.DefaultClusterJoinManager] checkNodeAuthenticationEnabled CONNECT(192.168.56.30:58883 -> 192.168.56.30:5701): Node authentication failed: Cluster authentication failed. Please make sure all members share the same value for 'confluence.cluster.name' and 'confluence.cluster.authentication.secret' in confluence.cfg.xml. 
2022-07-06 09:22:35,734 WARN [hz.confluence.cached.thread-6] [hazelcast.interceptor.authenticator.DefaultClusterJoinManager] checkNodeAuthenticationEnabled ACCEPT(192.168.56.30:5801 <- 192.168.56.30:38658): Node authentication failed: Cluster authentication failed. Please make sure all members share the same value for 'confluence.cluster.name' and 'confluence.cluster.authentication.secret' in confluence.cfg.xml. 
2022-07-06 09:22:35,734 WARN [hz.confluence.cached.thread-6] [hazelcast.nio.tcp.TcpIpAcceptor] log [192.168.56.30]:5801 [testcluster1] [3.12.11] com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.NodeConnectionException: Cluster authentication failed. Please make sure all members share the same value for 'confluence.cluster.name' and 'confluence.cluster.authentication.secret' in confluence.cfg.xml.
com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.NodeConnectionException: Cluster authentication failed. Please make sure all members share the same value for 'confluence.cluster.name' and 'confluence.cluster.authentication.secret' in confluence.cfg.xml.
	at com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.DefaultClusterJoinManager.checkNodeAuthenticationEnabled(DefaultClusterJoinManager.java:68)
	at com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.DefaultClusterJoinManager.accept(DefaultClusterJoinManager.java:47)
	at com.atlassian.confluence.impl.cluster.hazelcast.interceptor.ClusterJoinSocketInterceptor.onAccept(ClusterJoinSocketInterceptor.java:49)
	at com.hazelcast.nio.NodeIOService.interceptSocket(NodeIOService.java:300)
	at com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread.configureAndAssignSocket(TcpIpAcceptor.java:316)
	at com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread.access$1400(TcpIpAcceptor.java:138)
	at com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread$1.run(TcpIpAcceptor.java:305)
	at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
	at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
	at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
      atlassian-synchrony.log
      2022-07-06 09:22:20,784 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 2022-07-05 23:22:20,783 INFO [hz._hzInstance_1_testcluster1-Synchrony.cached.thread-8] [hazelcast.interceptor.authenticator.DefaultClusterJoinManager] enableNodeAuthentication : true
2022-07-06 09:22:20,784 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 2022-07-05 23:22:20,783 WARN [hz._hzInstance_1_testcluster1-Synchrony.cached.thread-8] [hazelcast.interceptor.authenticator.DefaultClusterJoinManager] ACCEPT(192.168.56.30:5701 <- 192.168.56.30:58883): Node authentication failed: Cluster authentication failed. Please make sure all members share the same value for cluster name and shared secret. 
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 2022-07-05 23:22:20,784 ERROR [hz._hzInstance_1_testcluster1-Synchrony.cached.thread-8] [synchrony.logging] uncaught exception {:throwable #error {
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	 :cause "org/apache/commons/lang3/StringUtils"
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	 :via
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	 [{:type java.lang.NoClassDefFoundError
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	   :message "org/apache/commons/lang3/StringUtils"
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	   :at [com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.NodeConnectionException getMessage "NodeConnectionException.java" 35]}]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	 :trace
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	 [[com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.NodeConnectionException getMessage "NodeConnectionException.java" 35]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread configureAndAssignSocket "TcpIpAcceptor.java" 320]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread access$1400 "TcpIpAcceptor.java" 138]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread$1 run "TcpIpAcceptor.java" 305]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker run "CachedExecutorServiceDelegate.java" 227]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [java.util.concurrent.ThreadPoolExecutor runWorker nil -1]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [java.util.concurrent.ThreadPoolExecutor$Worker run nil -1]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [java.lang.Thread run nil -1]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.util.executor.HazelcastManagedThread executeRun "HazelcastManagedThread.java" 64]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.util.executor.HazelcastManagedThread run "HazelcastManagedThread.java" 80]]}}

      This is reproducible on Data Center: Yes

      Steps to Reproduce

      1. Install Confluence DC as Clustered Type (one node is enough to reproduce issue) on Linux
      2. Choose multicast for the Node discovery
      3. Complete the install
      4. Create, edit and publish a single page
      5. tail the atlassian-confluence.log and atlassian-synchrony.log

      Expected Results

      • No warnings about Hazelcast when the Cluster discovery is "healthy".

      Actual Results

      Wait about 5 - 10 minutes, the below warnings are continually thrown in the atlassian-confluence.log and atlassian-synchrony.log:

      • Specifically, Confluence logs are referencing Synchrony 5701 hazelcast port
      • Synchrony logs are referencing Confluence Synchrony 5801 hazelcast port
      atlassian-confluence.log
      2022-07-06 09:22:20,785 WARN [hz.confluence.cached.thread-4] [hazelcast.interceptor.authenticator.DefaultClusterJoinManager] checkNodeAuthenticationEnabled CONNECT(192.168.56.30:58883 -> 192.168.56.30:5701): Node authentication failed: Cluster authentication failed. Please make sure all members share the same value for 'confluence.cluster.name' and 'confluence.cluster.authentication.secret' in confluence.cfg.xml. 
2022-07-06 09:22:35,734 WARN [hz.confluence.cached.thread-6] [hazelcast.interceptor.authenticator.DefaultClusterJoinManager] checkNodeAuthenticationEnabled ACCEPT(192.168.56.30:5801 <- 192.168.56.30:38658): Node authentication failed: Cluster authentication failed. Please make sure all members share the same value for 'confluence.cluster.name' and 'confluence.cluster.authentication.secret' in confluence.cfg.xml. 
2022-07-06 09:22:35,734 WARN [hz.confluence.cached.thread-6] [hazelcast.nio.tcp.TcpIpAcceptor] log [192.168.56.30]:5801 [testcluster1] [3.12.11] com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.NodeConnectionException: Cluster authentication failed. Please make sure all members share the same value for 'confluence.cluster.name' and 'confluence.cluster.authentication.secret' in confluence.cfg.xml.
com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.NodeConnectionException: Cluster authentication failed. Please make sure all members share the same value for 'confluence.cluster.name' and 'confluence.cluster.authentication.secret' in confluence.cfg.xml.
	at com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.DefaultClusterJoinManager.checkNodeAuthenticationEnabled(DefaultClusterJoinManager.java:68)
	at com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.DefaultClusterJoinManager.accept(DefaultClusterJoinManager.java:47)
	at com.atlassian.confluence.impl.cluster.hazelcast.interceptor.ClusterJoinSocketInterceptor.onAccept(ClusterJoinSocketInterceptor.java:49)
	at com.hazelcast.nio.NodeIOService.interceptSocket(NodeIOService.java:300)
	at com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread.configureAndAssignSocket(TcpIpAcceptor.java:316)
	at com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread.access$1400(TcpIpAcceptor.java:138)
	at com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread$1.run(TcpIpAcceptor.java:305)
	at com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker.run(CachedExecutorServiceDelegate.java:227)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
	at com.hazelcast.util.executor.HazelcastManagedThread.executeRun(HazelcastManagedThread.java:64)
	at com.hazelcast.util.executor.HazelcastManagedThread.run(HazelcastManagedThread.java:80)
      atlassian-synchrony.log
      2022-07-06 09:22:20,784 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 2022-07-05 23:22:20,783 INFO [hz._hzInstance_1_testcluster1-Synchrony.cached.thread-8] [hazelcast.interceptor.authenticator.DefaultClusterJoinManager] enableNodeAuthentication : true
2022-07-06 09:22:20,784 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 2022-07-05 23:22:20,783 WARN [hz._hzInstance_1_testcluster1-Synchrony.cached.thread-8] [hazelcast.interceptor.authenticator.DefaultClusterJoinManager] ACCEPT(192.168.56.30:5701 <- 192.168.56.30:58883): Node authentication failed: Cluster authentication failed. Please make sure all members share the same value for cluster name and shared secret. 
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 2022-07-05 23:22:20,784 ERROR [hz._hzInstance_1_testcluster1-Synchrony.cached.thread-8] [synchrony.logging] uncaught exception {:throwable #error {
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	 :cause "org/apache/commons/lang3/StringUtils"
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	 :via
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	 [{:type java.lang.NoClassDefFoundError
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	   :message "org/apache/commons/lang3/StringUtils"
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	   :at [com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.NodeConnectionException getMessage "NodeConnectionException.java" 35]}]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	 :trace
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	 [[com.atlassian.confluence.impl.cluster.hazelcast.interceptor.authenticator.NodeConnectionException getMessage "NodeConnectionException.java" 35]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread configureAndAssignSocket "TcpIpAcceptor.java" 320]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread access$1400 "TcpIpAcceptor.java" 138]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.nio.tcp.TcpIpAcceptor$AcceptorIOThread$1 run "TcpIpAcceptor.java" 305]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.util.executor.CachedExecutorServiceDelegate$Worker run "CachedExecutorServiceDelegate.java" 227]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [java.util.concurrent.ThreadPoolExecutor runWorker nil -1]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [java.util.concurrent.ThreadPoolExecutor$Worker run nil -1]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [java.lang.Thread run nil -1]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.util.executor.HazelcastManagedThread executeRun "HazelcastManagedThread.java" 64]
2022-07-06 09:22:20,785 DEBUG [227:StdOutHandler [/opt/atlassian/confluence/jre/bin/java]] 	  [com.hazelcast.util.executor.HazelcastManagedThread run "HazelcastManagedThread.java" 80]]}}

      Workaround

      1. Shutdown the entire Confluence Cluster
      2. Create/edit the <confluence-shared-home>/synchrony-args.properties file
        • Add in this line: 
          cluster.join.multicast.port=54328
        • This will configure Synchrony hazelcast multicast discovery to work over port 54328 (Confluence hazelcast multicast will still work over the default port 54327)
        • This will now stop Confluence and Synchrony hazelcast discovery conflicting with each other and stop the above mentioned logging lines
      3. Start Confluence (one node at a time)

      Impact

      If the Confluence and Synchrony DC Cluster Nodes are discovering each other and are "healthy", the above warnings can be safely ignored.

      If you are seeing Collaborative Editing issues, refer to:

      Attachments

        Issue Links

          is related to

          Bug - A problem which impairs or prevents the functions of the product. CONFSERVER-79547 Synchrony stops working with "java.io.IOException: Too many open files" on Linux when specifically using multicast Hazelcast discovery

          • Highest - Our top priority issues
          • Closed
          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Activity

            People

              5339cdd01cf4 Jeffery Xie
              hlam@atlassian.com Eric Lam
              Votes:
              6 Vote for this issue
              Watchers:
              17 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: