When starting a node that has outdated indexes (meaning: not having the latest index operation in the cluster replicated), it is supposed to automatically restore the indexes from the node that registered the latest index operation. However, if that node is unavailable or if that node never wrote a row to replicatedindexoperation, this will fail and the node will not have the latest index updates. See related JRASERVER-66550
Copy indexes from another node that has healthy indexes (have updated the indexes with latest operation).
The index restore fails and the below message is registered on the logs of the node that claimed the index restore request.
- Setup a new JIRA instance, which will be node1;
- Create a new project and some issues.
- Setup another JIRA instance, which will be node2. This node will restore the index from node1. Do not create or modify any issues in this node.
- Setup yet another JIRA instance, which will be node3. This node might attempt to restore the index from node2. (it depends which node claims the request first)
- node2 will fail to create an index snapshot as there are no rows in replicatedindexoperation having node_id = node2.
Problem is node2 shouldn't reply to the index request since it doesn't satisfy sanity check, see JRASERVER-66550.
That will leave only node1 as a potential index provider, which fixes the problem.
On the new node in the cluster (node 3, in this case), manually copy the indexes from node 2:
- Go to Administration > System > Indexing;
- Under Copy the Search Index from another node, select node 2 and copy;
Make sure the indexes on the chosen node are healthy by looking at the Indexing health check under the Administration > System > Support Tools > Health Checks section of that node.
- Touch any issues at the newly created node, so it will pass sanity check and will be able to share index.
- Perform a re-index operation on every new node brought up.