[BAM-20630] Cannot start Bamboo Elastic agents in 6.10 if Spot was enabled before the upgrade

Type: Bug
Resolution: Fixed
Priority: Highest
Fix Version/s: 6.10.3
Affects Version/s: 6.10.2
Component/s: Elastic Bamboo
Labels:
None

Support reference count:
3
Symptom Severity:
Severity 2 - Major
Bug Fix Policy:
View Atlassian Server bug fix policy

Issue Summary

If Spot instances were enabled and you upgrade to Bamboo 6.10, Elastic Bamboo functionality breaks (can't start elastic agents) because of references to deprecated instance types in the Spot bidding configuration.

Steps to Reproduce

Start Bamboo 6.9.2
Enable Spot instances at Bamboo Administration >> Overview >> (Elastic Bamboo) Spot Instances
Shutdown Bamboo and upgrade to Bamboo 6.10
Try to start an Elastic Agent > Error
Try to edit Spot instance configuration > Error

Expected Results

Elastic Bamboo works.

Actual Results

Can't start Elastic Agents or edit the Spot Instance configuration.

The below exception is thrown in the atlassian-bamboo.log file and in the UI when visiting the Spot instances UI:

2019-09-18 05:10:11,645 ERROR [http-nio-8085-exec-9] [FiveOhOh] 500 Exception was thrown.
com.google.common.util.concurrent.UncheckedExecutionException: com.thoughtworks.xstream.converters.ConversionException:
...
	at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
	at com.atlassian.bamboo.bandana.BambooCachingBandanaPersister.retrieve(BambooCachingBandanaPersister.java:36)
	at com.atlassian.bamboo.bandana.DefaultBandanaPersister.retrieve(DefaultBandanaPersister.java:43)
	at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32)
	at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:24)
	at com.atlassian.bamboo.bandana.BambooBandanaManager.getValue(BambooBandanaManager.java:28)
	at com.atlassian.bamboo.agent.elastic.server.SpotInstanceConfigurationAccessorImpl.getSpotInstanceConfig(SpotInstanceConfigurationAccessorImpl.java:28)
	at com.atlassian.bamboo.ww2.actions.admin.elastic.ConfigureSpotInstancesAction.getSpotInstanceConfig(ConfigureSpotInstancesAction.java:91)
	at com.atlassian.bamboo.ww2.actions.admin.elastic.ConfigureSpotInstancesAction.read(ConfigureSpotInstancesAction.java:58)
....
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.thoughtworks.xstream.converters.ConversionException:
...
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:77)
...
	at com.atlassian.bamboo.bandana.BandanaPersisterImpl.getObjectFromValue(BandanaPersisterImpl.java:76)
	at com.atlassian.bamboo.bandana.BandanaPersisterImpl.retrieve(BandanaPersisterImpl.java:31)
	at com.atlassian.bamboo.bandana.BambooCachingBandanaPersister$1.load(BambooCachingBandanaPersister.java:28)
	at com.atlassian.bamboo.bandana.BambooCachingBandanaPersister$1.load(BambooCachingBandanaPersister.java:25)
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
	... 290 more
Caused by: java.lang.IllegalArgumentException: No enum constant com.amazonaws.services.ec2.model.InstanceType.R5d16xlarge
	at java.lang.Enum.valueOf(Enum.java:238)
	at com.thoughtworks.xstream.converters.enums.EnumConverter.unmarshal(EnumConverter.java:49)
	at com.thoughtworks.xstream.core.TreeUnmarshaller.convert(TreeUnmarshaller.java:72)
	... 347 more

The below is logged to atlassian-bamboo.log when trying to start an Elastic instance:

2019-09-18 05:41:37,390 INFO [http-nio-8085-exec-6] [AccessLogFilter] bamboo POST http://localhost:8085/admin/elastic/createElasticInstances.action 397788kb
2019-09-18 05:41:38,651 ERROR [http-nio-8085-exec-6] [ManageElasticInstancesAction] Failed to start elastic instances
com.atlassian.aws.AWSException: Error when starting a new instance
	at com.atlassian.bamboo.agent.elastic.server.ElasticFunctionalityFacadeImpl.startInstancesInternal(ElasticFunctionalityFacadeImpl.java:217)
	at com.atlassian.bamboo.agent.elastic.server.ElasticFunctionalityFacadeImpl.startupAgents(ElasticFunctionalityFacadeImpl.java:159)
...
	at com.atlassian.bamboo.ww2.actions.admin.elastic.ManageElasticInstancesAction.create(ManageElasticInstancesAction.java:110)
...
	at java.lang.Thread.run(Thread.java:748)
Caused by: com.google.common.util.concurrent.UncheckedExecutionException: com.thoughtworks.xstream.converters.ConversionException:
...
	at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
	at com.atlassian.bamboo.bandana.BambooCachingBandanaPersister.retrieve(BambooCachingBandanaPersister.java:36)
	at com.atlassian.bamboo.bandana.DefaultBandanaPersister.retrieve(DefaultBandanaPersister.java:43)
	at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32)
	at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:24)
	at com.atlassian.bamboo.bandana.BambooBandanaManager.getValue(BambooBandanaManager.java:28)
	at com.atlassian.bamboo.agent.elastic.server.SpotInstanceConfigurationAccessorImpl.getSpotInstanceConfig(SpotInstanceConfigurationAccessorImpl.java:28)
	at com.atlassian.bamboo.agent.elastic.server.ElasticInstanceManagerImpl.createRemoteElasticInstanceObject(ElasticInstanceManagerImpl.java:240)
	at com.atlassian.bamboo.agent.elastic.server.ElasticInstanceManagerImpl.newElasticAgent(ElasticInstanceManagerImpl.java:255)
	at com.atlassian.bamboo.agent.elastic.server.ElasticFunctionalityFacadeImpl.startInstancesInternal(ElasticFunctionalityFacadeImpl.java:196)
	... 298 more

Workaround

Option 1: Completely remove spot configuration.

This option involves completely deleting all spot related configuration from the Bamboo database.

If following this route, you will need to manually reconfigure your Spot bids.

Shutdown Bamboo
Take a Bamboo database backup before running any modifications
You may also choose to extract the stored configuration if you want to reference your old spot prices at any point from the SERIALIZED_DATA column:
```
SELECT * FROM BANDANA WHERE BANDANA_KEY = 'EC2_SPOT_INSTANCES_CONFIG';
```

Run the below SQL to delete the Spot config from the Bamboo database:

DELETE FROM BANDANA WHERE BANDANA_KEY = 'EC2_SPOT_INSTANCES_CONFIG';

Start Bamboo
Reconfigure your bids manually

Discovered while testing

BAM-20403 M3 instances not available in US-East-2 for Elastic Agents

Closed

mentioned in: Page Failed to load; Page Failed to load

Krystian Brazulewicz made changes - 16/Jul/2020 6:38 AM

Remote Link

Original: This issue links to "Page (Confluence)" [ 452713 ]

Krystian Brazulewicz made changes - 16/Jul/2020 6:32 AM

Remote Link

New: This issue links to "Page (Confluence)" [ 494562 ]

Oleksandr Tkachenko made changes - 22/Oct/2019 12:37 PM

Remote Link

New: This issue links to "Page (Confluence)" [ 456497 ]

Marcin Walerianczyk made changes - 30/Sep/2019 8:01 AM

Resolution		New: Fixed [ 1 ]
Status	Original: Waiting for Release [ 12075 ]	New: Closed [ 6 ]

Bugfix Automation Bot made changes - 28/Sep/2019 10:00 AM

Support reference count

Original: 2

New: 3

Bugfix Automation Bot made changes - 27/Sep/2019 10:00 AM

Support reference count

Original: 1

New: 2

Krystian Brazulewicz made changes - 26/Sep/2019 1:57 PM

Remote Link

New: This issue links to "Page (Confluence)" [ 452713 ]

Alexey Chystoprudov made changes - 20/Sep/2019 9:37 AM

Status

Original: In Review [ 10051 ]

New: Waiting for Release [ 12075 ]

Alexey Chystoprudov made changes - 18/Sep/2019 10:09 AM

Fix Version/s

New: 6.10.3 [ 89405 ]

Alexey Chystoprudov made changes - 18/Sep/2019 10:09 AM

Status

Original: In Progress [ 3 ]

New: In Review [ 10051 ]

Assignee:: Alexey Chystoprudov

Reporter:: Jeremy Owen

Affected customers:: 4 This affects my team

Watchers:: 5 Start watching this issue

Created:: 18/Sep/2019 5:25 AM

Updated:: 16/Jul/2020 6:38 AM

Resolved:: 30/Sep/2019 8:01 AM

Details

Description

Issue Summary

Steps to Reproduce

Expected Results

Actual Results

Workaround

Option 1: Completely remove spot configuration.

Attachments

Issue Links

Forms

Activity

People

Dates