Full foreground reindex is not replicated to an offline node

Issue Summary

Some configuration changes, e.g. custom field searcher change, require reindexing all issues. If a full foreground reindex is run on one node, all other nodes pick up the index snapshot at the end of the operation.

Due to this bug the nodes that were offline while the full reindex was run will not pick up the index snapshot when they start. In effect, these nodes might have broken (outdated) index.

This is only reproducible on Data Center.

Steps to Reproduce

1. Set up a cluster with 2 nodes.
2. Start both nodes.
3. Create a custom field with no searcher and add it to the default screen.
4. Create an issue and set some value to the custom field.
5. Add a searcher to the custom field.
6. At this point this issue is not findable by the custom field value.
7. Shut down node 2.
8. Run full foreground reindex on node 1.
9. Start node 2.

Expected Results

Both nodes can find the issue by the custom field value.

Actual Results

Node 1 can find the issue by the custom field value.

Node 2 cannot find the issue by the custom field value, because it didn't pick up the full reindex result.

Workaround

When workaround is needed

You need the workaround only if you run a full foreground reindex while some other node is temporarily offline. New nodes joining the cluster for the first time won't be affected.

Workaround 1

Ensure there is an index snapshot in the shared home directory. It should have been produced either by the full foreground reindex or by the scheduled job (by default running daily).

Start the node that was offline during the foreground reindex with -Dcom.atlassian.jira.startup.rebuild.local.index=false.

More details about this property can be found here: https://confluence.atlassian.com/jirakb/failed-getting-index-on-start-1141970837.html

Workaround 2

Ensure there is an index snapshot in the shared home directory. It should have been produced either by the full foreground reindex or by the scheduled job (by default running daily).

Prior to starting the node that was offline during the foreground reindex, manually remove its local index (i.e. remove all files from $JIRAHOME/caches/indexesV2).

Workaround effect

Both workarounds will force the node that was temporarily offline to pick up an index snapshot from the shared home directory.