When starting a new node that is not listed as a peer on the existing nodes, it is added to the cluster and does not trigger a cluster panic. This behavior contradicts the information from the doc, which leads to believe the list needs to be updated on all nodes:
If the discovery mode is set to TCP/IP, you’ll need to update the confluence.cluster.peers property in the confluence.cfg.xml file for each node so the file lists all nodes in your cluster:
- Confluence DC clustered
- 2 or more nodes
- Create a three-node cluster with all members listed as peers:
- Stop all the nodes and remove the IP of one of the servers from the peers list of all members:
- Start the nodes one by one, keeping the removed member as the last one to be started
The nodes listed as peers are able to join the cluster. The node removed from the list cannot join the cluster unless the list of peers is updated as described in the documentation.
The member excluded from the list is able to join the cluster without issues. From the example above:
Looking at the TCP IP configuration printed on the logs, we can confirm that member 10.232.39.129 is not on the list:
We can also confirm the three members are part of the cluster on the Clustering management page.
Edit the cluster name on confluence.cfg.xml if you would like to isolate nodes that should not be part of the same cluster:
When the cluster name does not match, the node can't join the cluster and a panic is triggered on it, even if it has the list of current members on its peers configuration.
The behavior is only reproducible when starting the node that is not listed a member after the other nodes are already running. Also, that new node needs to have the other members in its own list of peers to find the cluster.
This actually matches what is described in the Hazelcats documentation since the mechanism is intended for node discovery, not for control or security purposes:
Note that all of the cluster members don't have to be listed there but at least one of them has to be active in cluster when a new member joins
We need to update the Confluence document, stating that it is recommended to update the peers list, but not mandatory.