Issue Summary

This is reproducible on Data Center: yes

Steps to Reproduce

Do either of these

• Start reindexing of two project concurrently
• Start background and project reindex concurrently
• Start project or background reindex so that it finishes at the time a scheduled snapshot is created
• Archive two projects concurrently
• Start two nodes concurrently when there's no fresh index snapshot in shared home <-- Not true, because there's a cluster lock preventing that

It has been always possible to create two index snapshots concurrently (full reindex + scheduled), but with recent Jira versions we have started creating snapshots more often.
Starting from 9.1.0 we create index snapshots on node start (if a fresh one doesn't exist).
Starting from 9.4.1 we create index snapshots after project reindex and project archive.

Expected Results

The minimum number of snapshots is created.
The snapshot is a backup of a healthy index.

Actual Results

Multiple snapshots are created concurrently.
Each snapshot can contain only part of the healthy index (in case of two project reindexes on two different nodes).

Workaround

1. Only run a second project reindex when the first one finishes.
2. Only archive a second project once the index snapshot is created after the first one has been archived (this can only be seen in the logs, search for "Done creating index snapshot")
3. Do not run project and background reindex at the same time.
4. Do not run project or background index around the time the scheduled snapshot is created