-
Bug
-
Resolution: Fixed
-
High
-
None
-
4
-
Severity 3 - Minor
-
1
-
Summary
Occasionally Elastic Agent creation calls fail with the error Invalid Base64 encoding of user data
Expected Results
All elastic Agent calls succeed with creation of a new Elastic Agent
Actual Results
Occasionally the creation attempt will fail:
The below exception is thrown in the xxxxxxx.log file:
com.amazonaws.services.ec2.model.AmazonEC2Exception: Invalid BASE64 encoding of user data (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: 24be1899-d089-48a5-b658-559d39a2034b)
Workaround
When failed, a new elastic agent call is made and the new agent starts normally.
Notes
it looks like when we use ava.util.Base64.getMimeEncoder().encodeToString(userData) to encode the userData for the instance (specifically the keypair generated per-instance) it is occasionally encoded with extra carriage returns which prevent proper decoding by AWS API.
- is cloned as
-
BDEV-15096 Loading...
[BAM-19945] Elastic Bamboo - Invalid Base64 encoding of user data
Hi @ezeidan,
It looks like requesting spot instances has been fixed in Bamboo 6.9.0
Thanks
Are their any known workarounds that don't require switching to on-demand instances or upgrading?
Marcos Rafael Kaissi Barbosa
added a comment - OK, thank you @Elias Zeidan. Hi angel16113731,
Thank you for your update. Please let me know if you see any issues with this in Bamboo 6.9.0.
Thanks!
Elias
Hi kaissi322311625,
Thank you very much for your update. Since the fix for this issue is in Bamboo 6.9.0, please upgrade to that version and try the Spot instances.
Thanks,
Elias
Hi @Elias Zeidan,
We are using 6.8.0 currently. I'll provide more feedback ASAP. I just requested the upgraded to 6.9.0
Cheers,
Marcos Rafael Kaissi Barbosa
added a comment - Hi,
I'm using version 5.14.1 and its unable to spin up spot instances.
Thanks
Marcos Rafael Kaissi Barbosa
added a comment - Hi,
I'm using version 5.14.1 and its unable to spin up spot instances.
Thanks Thank you for your comment! Can you confirm the version of Bamboo that you are using? Is it 6.9.0?
Thanks,
Elias | Bamboo Support
Hello florian.baumann1 and angel16113731,
Thank you for your comments!
Could you please confirm whether you are seeing this on the new Bamboo version, 6.9.0?
Thanks,
Elias | Bamboo Support
Hello.
Spawning Spot Instaces still shows the error described. Please consider reopening this ticket!
com.amazonaws.services.ec2.model.AmazonEC2Exception: Invalid BASE64 encoding of user data (Service: AmazonEC2; Status Code: 400; Error Code: InvalidParameterValue; Request ID: 4bdd2c74-6bad-46f5-96f9-2f9b09681c22)
Hello again. Is there any update on this bug? We're still unable to use spot instances properly, which is hindering our build team and increasing our EC2 costs.
Also, please consider raising the priority. Disabling a feature that a customer wants to use shouldn't be considered an appropriate workaround.
Today almost all of our elastic instances got this error during boot so we ran out of instances and had to shut off Spot Instances all together so it seems like this problem has gotten a lot worse all of a sudden.
Running into the same issue. Using Spot instances. Thank you for your attention.
dan
added a comment - Also agreed that it would be good to increase the severity of this bug. We are experiencing the same error and it looks to be happening more with Spot instances than OnDemand.
dan
added a comment - Also agreed that it would be good to increase the severity of this bug. We are experiencing the same error and it looks to be happening more with Spot instances than OnDemand.
Agree with the above comment that the severity for this bug should be increased. The current workaround of simply waiting for new agents to start up properly is not acceptable, especially for high-volume build environments. This bug essentially breaks elastic Bamboo for us.
Can we please up the severity on this issue? This can essentially cause a full outage of Bamboo until manual intervention is performed and even then it seems to take several attempts until it randomly is able to launch elastic instances.
Nick Burglin
added a comment - Here is the response I received from AWS Support while working with them for more details:
"It looks like there are newlines in the base64 content for the failed requests and none in the successful requests. The intermittent behavior is probably due to a difference in the way the base64 encoded data is being supplied to their automation tools. Perhaps it might be possible that the user-data is supplied using the base64 command line tool which outputs more human-readable output and pasting that output in with the line breaks. Basically omitting any line breaks from the base64 encoded text would ensure maximum compatibility."
Nick Burglin
added a comment - Here is the response I received from AWS Support while working with them for more details:
"It looks like there are newlines in the base64 content for the failed requests and none in the successful requests. The intermittent behavior is probably due to a difference in the way the base64 encoded data is being supplied to their automation tools. Perhaps it might be possible that the user-data is supplied using the base64 command line tool which outputs more human-readable output and pasting that output in with the line breaks. Basically omitting any line breaks from the base64 encoded text would ensure maximum compatibility."
I think Amazon fixed something, because spot instances in Bamboo 5.14.1 is working again.