We use AWS CloudFormation to run an internal JDC instance. When we upgrade, nodes are torn down and new ones are created. They are meant to recover the index from a snapshot in the shared-home directory.

We upgraded this morning, spinning up 1 node and waiting for it to serve on /status, then spinning up another node.

Timeline:

• 07:30: Node 1 spun up, upgrade commences
• 08:00: Node 2 spun up
• 08:01: Node 2 begins to recover the index, and completes in 30s (a successful recovery should take around 15 minutes for this instance)
• 08:35: We kick off a manual restore from index snapshot on Node 1
• 09:10: We realise Node 2's index is corrupt, despite a 'successful' recovery, and we kick off another restore from snapshot on Node 2
• 09:24: Index recovery completes and the indexes are both working now

As you can see from the attached log, Node 1 did not attempt index recovery after startup at all. Node 2 did, but it seems to have synced an empty index from Node 1.

NB: I confirmed that the snapshots were accessible by Jira, because we used them to recover the indexes manually.