Summary

Investigating index corruption we suspect caused by https://jira.atlassian.com/browse/JRASERVER-61099 it was discovered the portalpage and searchrequest indexes are not replicated properly after a node finishes a re-index

Environment

• Jira Data Center

Steps to Reproduce

1. Setup JIRA DC with 2 nodes, add some issues, some shared dashboards, some issue filters
2. Shut down node 2 and remove all contents under portalpage and searchrequest in /caches/indexes/entities ( to simulate https://jira.atlassian.com/browse/JRASERVER-61099 )
3. Startup node 2
4. Notice that searching for issues on node 2 works but searching for dashboards or filters does not, no results
5. Perform a reindex on node 1

   018-01-08 17:11:40,609 IssueIndexer:thread-3 INFO admin 1030x83x1 1jzv5oa 10.255.2.1 /rest/api/2/reindex [c.a.j.r.v2.index.ReindexResource] Re-indexing is 99% complete. Current index: Issue
   2018-01-08 17:11:43,029 JiraTaskExectionThread-2 INFO admin 1030x83x1 1jzv5oa 10.255.2.1 /rest/api/2/reindex [c.a.j.index.ha.DefaultIndexCopyService] Index backup started. Requesting node: node1
   2018-01-08 17:11:49,438 JiraTaskExectionThread-2 INFO admin 1030x83x1 1jzv5oa 10.255.2.1 /rest/api/2/reindex [c.a.j.index.ha.DefaultIndexCopyService] Index backup complete. Snapshot file: IndexSnapshot_10101.zip
   2018-01-08 17:11:49,444 JiraTaskExectionThread-2 INFO admin 1030x83x1 1jzv5oa 10.255.2.1 /rest/api/2/reindex [c.a.j.r.v2.index.ReindexResource] Re-indexing is 99% complete. Current index: SearchRequest
   2018-01-08 17:11:49,451 JiraTaskExectionThread-2 INFO admin 1030x83x1 1jzv5oa 10.255.2.1 /rest/api/2/reindex [c.a.j.r.v2.index.ReindexResource] Re-indexing is 99% complete. Current index: PortalPage
   2018-01-08 17:11:49,455 JiraTaskExectionThread-2 INFO admin 1030x83x1 1jzv5oa 10.255.2.1 /rest/api/2/reindex [c.a.j.r.v2.index.ReindexResource] Re-indexing is 100% complete. Current index: PortalPage
   2018-01-08 17:11:49,455 JiraTaskExectionThread-2 INFO admin 1030x83x1 1jzv5oa 10.255.2.1 /rest/api/2/reindex [c.a.j.r.v2.index.ReindexResource] Re-indexing finished
   

   • You'll see node 2 consume this index snapshot in its logs. Try searching for dashboards or filters again on node 2, still nothing
6. On node 2, perform an index copy from node 1
   • You should see dashboards and filters in search results once more
   • However, files are still missing in the directories

Expected Results

portalpage and searchrequest indexes are copied to the filesystem after restoring an index snapshot generated after a re-index

Actual Results

portalpage and searchrequest indexes are only copied during a index copy and even then the files themselves are missing

Workaround

• Perform an index copy to immediately restore portalpage and searchrequest indexes
• During the next maintenance window shut down all nodes and copy the portalpage and searchrequest directories from the node that completed a re-index to all other nodes