Issue Summary

When adding new node to Confluence cluster, the new node starts to recover the index when there is no index files in that node. This case is always happen when Confluence cluster runs in AWS Quick template, because adjusting nodes via AWS Auto Scaling.

There is no enough technical specification document in public, and the mechanism of Index Recovery and Keeping Index fresh is very very complicated now, Confluence DC system admins are hard to find the root cause when Index Recovery failed.

It would be useful for customers if Atlassian provide the public document regarding detail of Index Recovery mechanism and specification in cluster.

This is reproducible on Data Center: yes

Expected Results

Provide the public document regarding the detail of Index Recovery mechanism and specification in cluster.

ex:
===
1.Check if recovery required
2.Attempt recovery from a snapshot in shared home
3.Attempt recovery from a snapshot from another node
4.Rebuild index from DB
===

Actual Results

The following overall description is existing, but System Admin wants to know more details and the technical condition when the next step will be triggered.

Index recovery in a cluster

If you run Confluence in a cluster, a snapshot of the search index is also stored in the shared home directory. These snapshots are created by the Clean Journal Entries scheduled job which, by default, runs once per day.

When you start a Confluence node it will check whether its index is current, and if not, it will request a recovery snapshot from the shared home directory. If a snapshot is not available, it will generate a snapshot from a running node (with a matching build number). Once the recovery snapshot is extracted into the index directory, Confluence will continue the startup process. The journal service will then make any further updates required to bring the index up to date.

If the snapshot can't be generated, or is not received in time, existing index files will be removed and Confluence will perform a reindex on that node. If your index is very large or your file system slow, you may need to increase the time Confluence waits for the snapshot to be generated using the confluence.cluster.index.recovery.generation.timeout system property.

Index recovery only happens on node startup, so if you suspect a problem with a particular cluster node index, restart that node to trigger index recovery.

The index recovery snapshot isn't used when you manually rebuild your index from the UI. The rebuild process generates a brand new snapshot, before propagating it to other nodes in the cluster.

Workaround

Currently there is no known workaround for this behavior. A workaround will be added here when available