Index recovery does not run on startup automatically if a working index does not exist

We use AWS CloudFormation to run an internal JDC instance. When we upgrade, nodes are torn down and new ones are created. They are meant to recover the index from a snapshot in the shared-home directory.

We upgraded this morning, spinning up 1 node and waiting for it to serve on /status, then spinning up another node.

Timeline:

• 07:30: Node 1 spun up, upgrade commences
• 08:00: Node 2 spun up
• 08:01: Node 2 begins to recover the index, and completes in 30s (a successful recovery should take around 15 minutes for this instance)
• 08:35: We kick off a manual restore from index snapshot on Node 1
• 09:10: We realise Node 2's index is corrupt, despite a 'successful' recovery, and we kick off another restore from snapshot on Node 2
• 09:24: Index recovery completes and the indexes are both working now

As you can see from the attached log, Node 1 did not attempt index recovery after startup at all. Node 2 did, but it seems to have synced an empty index from Node 1.

NB: I confirmed that the snapshots were accessible by Jira, because we used them to recover the indexes manually.