Jira takes snapshot of indexes on a node where full foreground re-index is running.

Issue Summary

While running a Foreground Re-index on a node, Jira creates an Index Snapshot on the same Node.

Steps to Reproduce:

• Start 2 Nodes with empty Indexes

• On Node 1, initiate a Full-Lock Re-index:

  2022-01-06 12:02:51:,958+0100 JiraTaskExecutionThread-1 INFO xkatoma 722x489x1 11q1bh5 10.210.8.117,10.120.14.166 /secure/admin/IndexReIndex!reindex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 0% complete. Current index: Issue

• Create a IndexSnapshot Schedule to run every min:
  

• When Reindex is at 14% :

  2022-01-06 12:22:14,296+0100 IssueIndexer:thread-18 INFO xkatoma 722x489x1 11q1bh5 10.210.8.117,10.120.14.166 /secure/admin/IndexReIndex!reindex.jspa [c.a.j.w.a.admin.index.IndexAdminImpl] Re-indexing is 14% complete. Current index: Issue

• An IndexSnapshot_16000 was created on Node 1:

  2022-01-06 12:22:40,569+0100 ClusterMessageHandlerServiceThread:thread-1 INFO      [c.a.j.index.ha.DefaultIndexCopyService] Received message: "Backup Index" - request to create index snapshot from node: node2 on current node: node1
  2022-01-06 12:22:40,589+0100 ClusterMessageHandlerServiceThread:thread-1 INFO      [c.a.j.index.ha.DefaultIndexCopyService] Index backup started. Requesting node: node2, currentNode: node1
  
  2022-01-06 12:22:40,655+0100 ClusterMessageHandlerServiceThread:thread-1 INFO      [c.a.j.index.ha.IndexUtils] Creating index snapshot of type: SNAPPY in local temporary directory: /opt/atl/jira-dc-uat4/current/temp/snapshot1641468160655/IndexSnapshot_16000.tar.sz

• From Node 2 logs, When Node 2 was started the Index consistency failed while checking local index on Node start, this can occur if there's no IndexSnapshot on the current node or the IndexSnapshot present in shared directory is not clean:

  2022-01-06 12:22:39,863+0100 localhost-startStop-1 INFO      [c.a.jira.startup.ClusteringLauncher] Checking local index on node start
  2022-01-06 12:22:40,354+0100 localhost-startStop-1 WARN      [c.a.j.issue.index.IndexConsistencyUtils] Index consistency check failed for index 'Issue': expectedCount=4356988; actualCount=0
  2022-01-06 12:22:40,356+0100 localhost-startStop-1 INFO      [c.a.jira.cluster.DefaultClusterManager] Current node: node2 index can't be rebuilt. Requesting an index from any other node. Current list of other nodes: [node1]

• So a request was sent to Node 1:

  2022-01-06 12:22:40,515+0100 localhost-startStop-1 INFO      [c.a.jira.cluster.DefaultClusterManager] Sending message: "Backup Index" - request to create index snapshot from node: ANY on current node: node2

• Node 2 was restored from IndexSnapshot created by Node 1:

  2022-01-06 12:23:08,926+0100 ClusterMessageHandlerServiceThread:thread-1 INFO      [c.a.j.index.ha.DefaultIndexCopyService] Received message: "Index Backed Up" - notification that node: node1 created an index snapshot which can be restored on current node: node2
  2022-01-06 12:23:08,929+0100 ClusterMessageHandlerServiceThread:thread-1 INFO      [c.a.j.index.ha.DefaultIndexCopyService] Index restore started. Total 0 issues on instance before loading Snapshot file: IndexSnapshot_16000.tar.sz

The reason for IndexSnapshot created on Node 1 was because there was a request from Node 2 and the IndexSnapshot was created is due to automatic index copy for which we cannot disable the automatic copy process. This is being worked under Suggestion Request: JRASERVER-66916 

But we do not expect an IndexSnapshot to be created while running a Reindex because creating a Snapshot of Invalid Index is not useful

Expected Results

Jira chooses different node or waits until full foreground re-index finishes.

Actual Results

Jira might start taking a snapshot of index on the node that is running the re-index.

Workaround

Do not run re-index during scheduled index snapshot taking.