Details
Medium Description
Issue Summary
This is reproducible on Data Center: yes
Steps to Reproduce
- Setup a 2 node cluster on Azure using a Bitbucket 8.9.0 distribution build using instructions here
- To see the error message below, add to your shared bitbucket.properties file:
logging.logger.com.hazelcast.azure=TRACE.
Tested on Bitbucket 7.21.15 (edit later: also tested on 7.21.18 and node discovery works) after the previous upgrade to hazelcast-azure, and node discovery worked successfully, so likely an upgrade between 7.21.15 and 8.9.0 has broken azure node discovery.
Expected Results
Successful node discovery after starting both nodes up in Admin > Clustering.
Successful node discovery and joining a cluster should produce logs:
2023-10-17 02:45:58,014 INFO [hz.hazelcast.event-2] c.a.s.i.c.HazelcastClusterService Node '/10.2.0.4:5701' was ADDED to the cluster. Updated cluster: [/10.2.0.7:5701 master this uuid='a05bd62b-34b8-462f-91e5-8a3b2ae6d0af' vm-id='fd291b06-75a0-4cfd-aa00-5f3c9a6b5df9'], [/10.2.0.4:5701 uuid='0ecacc90-e282-436d-a2f7-25a8fee9051b' vm-id='f9b16fb0-86f7-4b0f-9e9a-04240185f3e1']
Actual Results
The below exception is thrown in the atlassian-bitbucket.log file when starting one of the nodes during the Hazelcast initialisation stage.
2023-10-17 05:23:02,891 TRACE [spring-startup] c.h.azure.AzureDiscoveryStrategy Failed to discover nodes with Azure SPI java.lang.RuntimeException: java.io.IOException: java.lang.NoClassDefFoundError: com/nimbusds/oauth2/sdk/http/CommonContentTypes at rx.exceptions.Exceptions.propagate(Exceptions.java:57) at rx.observables.BlockingObservable.blockForSingle(BlockingObservable.java:463) at rx.observables.BlockingObservable.single(BlockingObservable.java:340) at com.microsoft.azure.management.compute.implementation.VirtualMachinesInner.listByResourceGroup(VirtualMachinesInner.java:1505) at com.microsoft.azure.management.resources.fluentcore.arm.collection.implementation.TopLevelModifiableResourcesImpl.listByResourceGroup(TopLevelModifiableResourcesImpl.java:123) at com.hazelcast.azure.AzureDiscoveryStrategy.discoverVMs(AzureDiscoveryStrategy.java:147) at com.hazelcast.azure.AzureDiscoveryStrategy.discoverNodes(AzureDiscoveryStrategy.java:96) at com.hazelcast.azure.AzureDiscoveryStrategy.discoverLocalMetadata(AzureDiscoveryStrategy.java:83) at com.hazelcast.spi.discovery.impl.DefaultDiscoveryService.discoverLocalMetadata(DefaultDiscoveryService.java:88) at com.hazelcast.instance.Node.mergeEnvironmentProvidedMemberMetadata(Node.java:600) at com.hazelcast.instance.Node.start(Node.java:445) at com.hazelcast.instance.HazelcastInstanceImpl.<init>(HazelcastInstanceImpl.java:136) at com.hazelcast.instance.HazelcastInstanceFactory.constructHazelcastInstance(HazelcastInstanceFactory.java:203) at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:182) at com.hazelcast.instance.HazelcastInstanceFactory.newHazelcastInstance(HazelcastInstanceFactory.java:132) at com.hazelcast.core.Hazelcast.newHazelcastInstance(Hazelcast.java:57) at com.atlassian.stash.internal.hazelcast.HazelcastFactoryBean.newInstance(HazelcastFactoryBean.java:126) at com.atlassian.stash.internal.hazelcast.HazelcastFactoryBean.createInstance(HazelcastFactoryBean.java:66) at com.atlassian.stash.internal.hazelcast.HazelcastFactoryBean.createInstance(HazelcastFactoryBean.java:34) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:920) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:583) at javax.servlet.GenericServlet.init(GenericServlet.java:158) at java.base/java.lang.Thread.run(Thread.java:829) ... 34 frames trimmed
Workaround
Currently there is no known workaround for this behavior. A workaround will be added here when available