Bundled Elasticsearch indexes may need to be nuked and rebuild when upgrading Bitbucket Server to 7.1.x or newer, otherwise Elasticsearch will fail to start

Issue Summary

When upgrading Bitbucket Server with bundled Elasticsearch from 5.16.x straight to 7.1.x or newer, Elasticsearch indexes may need to be nuked and rebuilt from scratch because Elasticsearch will fail to upgradade them, which blocks Elasticsearch from starting up.
Elasticsearch logs report the following issue:

2020-04-14 14:40:54 Commons Daemon procrun stderr initialized
Exception in thread "main" tion: The index [[bitbucket-search-v1/4YpLxPxQQF2Hn268Mxv3ug]] was created with version [5.5.3] but the minimum compatible version is [6.0.0-beta1]. It should be re-indexed in Elasticsearch 6.x before upgrading to 7.5.2.
	at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.checkSupportedVersion(MetaDataIndexUpgradeService.java:113)
	at org.elasticsearch.cluster.metadata.MetaDataIndexUpgradeService.upgradeIndexMetaData(MetaDataIndexUpgradeService.java:87)
	at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:227)
	at org.elasticsearch.gateway.GatewayMetaState.upgradeMetaData(GatewayMetaState.java:167)
	at org.elasticsearch.gateway.GatewayMetaState.start(GatewayMetaState.java:90)
	at org.elasticsearch.node.Node.start(Node.java:696)
	at org.elasticsearch.bootstrap.Bootstrap.start(Bootstrap.java:293)
	at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:378)
	at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:159)
	at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:150)
	at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:86)
	at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:125)
	at org.elasticsearch.cli.Command.main(Command.java:90)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:115)
	at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92)
Refer to the log for complete error details.

NOTES:

• As can be seen, this turns out to be a bug or limitation in Elasticsearch, not in Bitbucket Server. Bitbucket Server itself is upgraded without any problems.
• The message suggests the index needs to be rebuilt by an Elasticsearch version 6.x before they can be upgraded to 7.5.2.
• Performing an intermediary upgrade of Bitbucket from 5.16.11 to 6.10 (which bundles Elasticsearch 6.8.6) does not help because Elasticsearch 6.8.6 will not upgrade the index from 5.5.3. As the message suggests, the index needs to be rebuilt in Elasticsearch 6.x.
• This issue does not affect customers whose first version of Bitbucket ever installed bundled Elasticsearch 6.x or later.

Steps to Reproduce

1. In a Windows machine, install Bitbucket Server version is 5.16.11, which bundles Elasticsearch 5.5.3, both of them starting up as Windows Service.
2. Add a repository and let Elasticsearch finish indexing it.
3. Shutdown Bitbucket Server and Elasticsearch services.
4. Upgrade that Bitbucket instance to 7.1.1, which bundles Elasticsearch 7.5.2
5. The upgrade scripts will update both Windows services as expected.
6. Notice that only the Bitbucket service will start automatically. Elasticsearch service will be stopped, won't start at all.
7. Take a look at the Elasticsearch logs and see that error message mentioned above.

Expected Results

• When upgrading from a Bitbucket Server version whose bundled Elasticsearch version is lower than 6.x, the Bitbucket installer should nuke the Elasticsearch caches before finishing the installation, so that in the last installer step when the services are started, both services do get started.
• We should make it clear in our upgrade guide that Elasticsearch caches may need to be nuked, and that rebuilding the index may take several hours, possibly resulting in some overhead on the server and search functionality showing incomplete results in the meantime.

Actual Results

1. The bundled Elasticsearch 7.5.2 fails to start when after being upgraded from 5.5.3.
2. Elasticsearch logs report the issue mentioned above.

Workaround

Manually delete the index by performing the following steps:

1. Stop Bitbucket Server and Elasticsearch services
2. Delete the contents of <bitbucket-home>/shared/search/data/nodes directory
3. Start Bitbucket Server and Elasticsearch services
4. Wait until the index is completely rebuilt for obtaining complete search results.