Uploaded image for project: 'Jira Server and Data Center'
  1. Jira Server and Data Center
  2. JRASERVER-62669

Automatic restore of indexes will fail if the node that registered the latest index operation is unavailable

    XMLWordPrintable

    Details

      Description

      Summary

      When starting a node that has outdated indexes (meaning: not having the latest index operation in the cluster replicated), it is supposed to automatically restore the indexes from the node that registered the latest index operation. However, if that node is unavailable or if that node never wrote a row to replicatedindexoperation, this will fail and the node will not have the latest index updates. See related JRASERVER-66550

      Expected Behavior

      Copy indexes from another node that has healthy indexes (have updated the indexes with latest operation).

      Actual Behavior

      The index restore fails and the below message is registered on the logs of the node that claimed the index restore request.

      2016-09-30 16:25:33,487 ClusterMessageHandlerServiceThread:thread-1 INFO      [jira.index.ha.DefaultIndexCopyService] Index backup started. Requesting node: node3
      2016-09-30 16:25:33,488 ClusterMessageHandlerServiceThread:thread-1 WARN      [jira.index.ha.DefaultIndexCopyService] Index backup failed - latest index operation not found. Requesting node: node3
      

      Steps to Reproduce

      1. Setup a new JIRA instance, which will be node1;
      2. Create a new project and some issues.
      3. Setup another JIRA instance, which will be node2. This node will restore the index from node1. Do not create or modify any issues in this node.
      4. Setup yet another JIRA instance, which will be node3. This node might attempt to restore the index from node2. (it depends which node claims the request first)
      5. node2 will fail to create an index snapshot as there are no rows in replicatedindexoperation having node_id = node2.

      Note

      Problem is node2 shouldn't reply to the index request since it doesn't satisfy sanity check, see JRASERVER-66550.
      That will leave only node1 as a potential index provider, which fixes the problem.

      Workaround

      option1

      On the new node in the cluster (node 3, in this case), manually copy the indexes from node 2:

      1. Go to Administration > System > Indexing;
      2. Under Copy the Search Index from another node, select node 2 and copy;

      Make sure the indexes on the chosen node are healthy by looking at the Indexing health check under the Administration > System > Support Tools > Health Checks section of that node.

      option2
      1. Touch any issues at the newly created node, so it will pass sanity check and will be able to share index.
      option3
      • Perform a re-index operation on every new node brought up.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              jpalharini Joao Palharini (Inactive)
              Votes:
              12 Vote for this issue
              Watchers:
              17 Start watching this issue

                Dates

                Created:
                Updated: