In the AWS CloudFormation and Azure worlds, the ephemeral nature of nodes means that index recovery is no longer a one off disaster recovery task unlikely to be performed very often, and is now performed every time a node is created. This happens on every upgrade, and often during incidents. For us on our dogfooding instances that is at least once a week for the upgrade.

Some suggestions for improvements to the procedure:

1. Allow a snapshot to be triggered with a button in the UI
2. Allow a snapshot to be recovered from by entering a filename in the UI, similar to Jira
3. Create a UI for pulling an index snapshot from a healthy node (or pushing to an unhealthy one), similar to Jira
4. Speed up the index recovery process. This currently seems to take 1 minute per GB of index - for us it takes 20 minutes to recover a 20gb index. Unsure if this is a linear correlation or exponential, but 20 minutes of extra downtime per node is not ideal
5. Allow index recovery to occur in the background so at least the system is up during recovery