Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-73364

Restoring an index snapshot relies on getNumberOfIssues(), which can prevent a corrective index snapshot restore to succeed

    XMLWordPrintable

Details

    Description

      Issue Summary

      During an index snapshot restore, Jira tries to obtain the count issues in the existing index during the restore process. If the existing index is corrupt, this process fails, which prevents the index snapshot restore from succeeding.

      This operation could be taken by an admin during a corrupt index remediation activity.

      Steps to Reproduce

      1. Corrupt the node's index
      2. Initiate a snapshot restore from a healthy index

      Expected Results

      The index snapshot restore ignores the corrupted index, and replaces it with the snapshot

      Actual Results

      Jira tries to open the corrupt index and fails to restore snapshot

      2022-03-06 20:39:46,579-0500 ClusterMessageHandlerServiceThread:thread-1 INFO      [c.a.j.index.ha.DefaultIndexCopyService] Received message: "Index Backed Up" - notification that node: HEALTHY_NODE created an index snapshot which can be restored on current node: CORRUPT_NODE
      2022-03-06 20:39:46,590-0500 ClusterMessageHandlerServiceThread:thread-1 ERROR      [c.a.jira.cluster.OfBizMessageHandlerService] There was a problem handling a cluster message
      com.atlassian.jira.issue.index.SearchUnavailableException: com.atlassian.jira.util.RuntimeIOException: org.apache.lucene.index.CorruptIndexException: file mismatch, expected id=cp14pmqikixkrdv3owtxwycs9, got=dp6lmolj0xv5fecjuw55j6iuo (resource=MMapIndexInput(path="F:\bin\Atlassian\Application Data\JIRA\caches\indexesV1\issues\_9p908_Lucene50_0.doc"))
      	at com.atlassian.jira.issue.index.ThreadLocalSearcherCache$Cache.retrieveEntitySearcher(ThreadLocalSearcherCache.java:139)
      	at com.atlassian.jira.issue.index.ThreadLocalSearcherCache.getSearcher(ThreadLocalSearcherCache.java:40)
      ...
      	at com.atlassian.jira.index.ha.DefaultIndexCopyService$MessageConsumer.getNumberOfIssues(DefaultIndexCopyService.java:166)
      	at com.atlassian.jira.index.ha.DefaultIndexCopyService$MessageConsumer.restoreIndex(DefaultIndexCopyService.java:153)
      	at com.atlassian.jira.index.ha.DefaultIndexCopyService$MessageConsumer.receive(DefaultIndexCopyService.java:182) 

      Workaround

      Delete the indexes from the node's caches\indexesV1: directory, then restart Jira to induce a snapshot restore:

      • changes
      • comments
      • entities
      • issues
      • plugins
      • worklogs

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              allewellyn@atlassian.com Alex [Atlassian,PSE]
              Votes:
              4 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: