Details
-
Bug
-
Resolution: Fixed
-
Highest
-
6.10.2
-
None
-
3
-
Severity 2 - Major
-
Description
Issue Summary
If Spot instances were enabled and you upgrade to Bamboo 6.10, Elastic Bamboo functionality breaks (can't start elastic agents) because of references to deprecated instance types in the Spot bidding configuration.
Steps to Reproduce
- Start Bamboo 6.9.2
- Enable Spot instances at Bamboo Administration >> Overview >> (Elastic Bamboo) Spot Instances
- Shutdown Bamboo and upgrade to Bamboo 6.10
- Try to start an Elastic Agent > Error
- Try to edit Spot instance configuration > Error
Expected Results
Elastic Bamboo works.
Actual Results
Can't start Elastic Agents or edit the Spot Instance configuration.
The below exception is thrown in the atlassian-bamboo.log file and in the UI when visiting the Spot instances UI:
2019-09-18 05:10:11,645 ERROR [http-nio-8085-exec-9] [FiveOhOh] 500 Exception was thrown. com.google.common.util.concurrent.UncheckedExecutionException: com.thoughtworks.xstream.converters.ConversionException: ... at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830) at com.atlassian.bamboo.bandana.BambooCachingBandanaPersister.retrieve(BambooCachingBandanaPersister.java:36) at com.atlassian.bamboo.bandana.DefaultBandanaPersister.retrieve(DefaultBandanaPersister.java:43) at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32) at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:24) at com.atlassian.bamboo.bandana.BambooBandanaManager.getValue(BambooBandanaManager.java:28) at com.atlassian.bamboo.agent.elastic.server.SpotInstanceConfigurationAccessorImpl.getSpotInstanceConfig(SpotInstanceConfigurationAccessorImpl.java:28) at com.atlassian.bamboo.ww2.actions.admin.elastic.ConfigureSpotInstancesAction.getSpotInstanceConfig(ConfigureSpotInstancesAction.java:91) at com.atlassian.bamboo.ww2.actions.admin.elastic.ConfigureSpotInstancesAction.read(ConfigureSpotInstancesAction.java:58) .... at java.lang.Thread.run(Thread.java:748) Caused by: com.thoughtworks.xstream.converters.ConversionException: ... at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:77) ... at com.atlassian.bamboo.bandana.BandanaPersisterImpl.getObjectFromValue(BandanaPersisterImpl.java:76) at com.atlassian.bamboo.bandana.BandanaPersisterImpl.retrieve(BandanaPersisterImpl.java:31) at com.atlassian.bamboo.bandana.BambooCachingBandanaPersister$1.load(BambooCachingBandanaPersister.java:28) at com.atlassian.bamboo.bandana.BambooCachingBandanaPersister$1.load(BambooCachingBandanaPersister.java:25) at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527) at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319) at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282) at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197) ... 290 more Caused by: java.lang.IllegalArgumentException: No enum constant com.amazonaws.services.ec2.model.InstanceType.R5d16xlarge at java.lang.Enum.valueOf(Enum.java:238) at com.thoughtworks.xstream.converters.enums.EnumConverter.unmarshal(EnumConverter.java:49) at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72) ... 347 more
The below is logged to atlassian-bamboo.log when trying to start an Elastic instance:
2019-09-18 05:41:37,390 INFO [http-nio-8085-exec-6] [AccessLogFilter] bamboo POST http://localhost:8085/admin/elastic/createElasticInstances.action 397788kb 2019-09-18 05:41:38,651 ERROR [http-nio-8085-exec-6] [ManageElasticInstancesAction] Failed to start elastic instances com.atlassian.aws.AWSException: Error when starting a new instance at com.atlassian.bamboo.agent.elastic.server.ElasticFunctionalityFacadeImpl.startInstancesInternal(ElasticFunctionalityFacadeImpl.java:217) at com.atlassian.bamboo.agent.elastic.server.ElasticFunctionalityFacadeImpl.startupAgents(ElasticFunctionalityFacadeImpl.java:159) ... at com.atlassian.bamboo.ww2.actions.admin.elastic.ManageElasticInstancesAction.create(ManageElasticInstancesAction.java:110) ... at java.lang.Thread.run(Thread.java:748) Caused by: com.google.common.util.concurrent.UncheckedExecutionException: com.thoughtworks.xstream.converters.ConversionException: ... at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830) at com.atlassian.bamboo.bandana.BambooCachingBandanaPersister.retrieve(BambooCachingBandanaPersister.java:36) at com.atlassian.bamboo.bandana.DefaultBandanaPersister.retrieve(DefaultBandanaPersister.java:43) at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32) at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:24) at com.atlassian.bamboo.bandana.BambooBandanaManager.getValue(BambooBandanaManager.java:28) at com.atlassian.bamboo.agent.elastic.server.SpotInstanceConfigurationAccessorImpl.getSpotInstanceConfig(SpotInstanceConfigurationAccessorImpl.java:28) at com.atlassian.bamboo.agent.elastic.server.ElasticInstanceManagerImpl.createRemoteElasticInstanceObject(ElasticInstanceManagerImpl.java:240) at com.atlassian.bamboo.agent.elastic.server.ElasticInstanceManagerImpl.newElasticAgent(ElasticInstanceManagerImpl.java:255) at com.atlassian.bamboo.agent.elastic.server.ElasticFunctionalityFacadeImpl.startInstancesInternal(ElasticFunctionalityFacadeImpl.java:196) ... 298 more
Workaround
Option 1: Completely remove spot configuration.
This option involves completely deleting all spot related configuration from the Bamboo database.
If following this route, you will need to manually reconfigure your Spot bids.
- Shutdown Bamboo
- Take a Bamboo database backup before running any modifications
- You may also choose to extract the stored configuration if you want to reference your old spot prices at any point from the SERIALIZED_DATA column:
SELECT * FROM BANDANA WHERE BANDANA_KEY = 'EC2_SPOT_INSTANCES_CONFIG'; - Run the below SQL to delete the Spot config from the Bamboo database:
DELETE FROM BANDANA WHERE BANDANA_KEY = 'EC2_SPOT_INSTANCES_CONFIG'; - Start Bamboo
- Reconfigure your bids manually
