Git checkout not working with AWS CodeCommit repository

Problem

When using an AWS CodeCommit repository, checkouts are failing with KeyExchange signature verification failed for key type=ssh-rsa

Environment

Bamboo 8, 9

Steps to Reproduce

1. Follow the instructions on AWS Documentation to configure GIT and SSH to connect to AWS CodeCommit
2. Save the Git repository
3. Test the repository or run a plan to test the checkout task

Expected Results

The checkout task should succeed

Actual Results

The checkout fails with KeyExchange signature verification failed for key type=ssh-rsa:

2022-09-12 13:16:24,125 WARN [sshd-SshClient[5c57312c]-nio2-thread-1] [ClientSessionImpl] exceptionCaught(ClientSessionImpl[APKASDQ4LWHL2EXAMPLE@git-codecommit.us-west-2.amazonaws.com/52.119.168.71:22])[state=Opened] SshException: KeyExchange signature verification failed for key type=ssh-rsa
org.apache.sshd.common.SshException: KeyExchange signature verification failed for key type=ssh-rsa
	at org.apache.sshd.client.kex.DHGEXClient.next(DHGEXClient.java:241)
	at org.apache.sshd.common.session.helpers.AbstractSession.handleKexMessage(AbstractSession.java:607)
	at org.apache.sshd.common.session.helpers.AbstractSession.doHandleMessage(AbstractSession.java:501)
	at org.apache.sshd.common.session.helpers.AbstractSession.handleMessage(AbstractSession.java:429)
	at org.apache.sshd.common.session.helpers.AbstractSession.decode(AbstractSession.java:1466)
	at org.apache.sshd.common.session.helpers.AbstractSession.messageReceived(AbstractSession.java:389)
	at org.apache.sshd.common.session.helpers.AbstractSessionIoHandler.messageReceived(AbstractSessionIoHandler.java:64)
	at org.apache.sshd.common.io.nio2.Nio2Session.handleReadCycleCompletion(Nio2Session.java:359)
	at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:336)
	at org.apache.sshd.common.io.nio2.Nio2Session$1.onCompleted(Nio2Session.java:333)
	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.lambda$completed$0(Nio2CompletionHandler.java:38)
	at java.base/java.security.AccessController.doPrivileged(Native Method)
	at org.apache.sshd.common.io.nio2.Nio2CompletionHandler.completed(Nio2CompletionHandler.java:37)
	at java.base/sun.nio.ch.Invoker.invokeUnchecked(Invoker.java:127)
	at java.base/sun.nio.ch.Invoker$2.run(Invoker.java:219)
	at java.base/sun.nio.ch.AsynchronousChannelGroupImpl$1.run(AsynchronousChannelGroupImpl.java:112)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)

Workaround

Per repository/User configuration

1. Modify the repository and set authentication to none. This will make Bamboo bypass the SSH-PROXY.
2. Set the following ~/.ssh/config file on every Agent (or Local agent)

   Host git-codecommit.*.amazonaws.com
     User APKASDQ4LWHL2EXAMPLE
     IdentityFile ~/.ssh/codecommit
   

3. The identify file ~/.ssh/codecommit need to be maintained by the administrator, or you can optionally add its contents to a variable in the Plan and then use a script task to create it before the checkout task

Generic CodeCommit configuration

1. This method is similar to the first one but allows you to have a "generic" ~/.ssh/config file as the user will be referenced from the repository string
2. Modify the repository and set authentication to none. This will make Bamboo bypass the SSH-PROXY.
3. On the repository configuration, set the user directly on the repository connection string. E.g. ssh://APKASDQ4LWHL2EXAMPLE@git-codecommit.us-west-2.amazonaws.com/v1/repos/test123
4. Set the following ~/.ssh/config file on every Agent (or Local agent)

   Host git-codecommit.*.amazonaws.com
     IdentityFile ~/.ssh/codecommit
   

5. The identify file ~/.ssh/codecommit needs to be maintained by the administrator, or you can optionally add its contents to a variable in the Plan and then use a script task to create the file and set its permissions before the checkout task

Use a Script task and plain git checkout

1. Do not configure a repository checkout task in Bamboo. It will be replaced by the script task
2. Follow the configuration steps provided by AWS on each Agent
3. Create and configure a script task that will run the git checkout

   #!/bin/bash -x
   # Your remote/local agent must have proper SSH keys exchanged with your Git repository
   # This script will not manage authentication
   
   # This code is untested and may need further adjustments to suit your requirements
   # Use this script only if after finding that enabling Repository Caching on your repositories 
   # does not suit you
   REPOSITORY=${bamboo.planRepository.repositoryUrl}
   REVISION=${bamboo.planRepository.revision}
   BRANCHNAME=${bamboo.planRepository.branchName}
   GIT=${bamboo.capability.system.git.executable}
   GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=no -o BatchMode=yes -o UserKnownHostsFile=/dev/null"
   
   ${GIT} init
   ${GIT} config remote.origin.url >&- || ${GIT} remote add origin ${REPOSITORY}
   
   # [0] = REVISION
   # [1] = Branch/HEAD
   CUSTOM_GIT_HEADS=($(${GIT} ls-remote --heads ${REPOSITORY}))
   
   ${GIT} fetch ${REPOSITORY} +${CUSTOM_GIT_HEADS[1]}:${CUSTOM_GIT_HEADS[1]} --update-head-ok --progress --verbose
   ${GIT} show-ref ${BRANCHNAME}
   

4. The identify file ~/.ssh/codecommit needs to be maintained by the administrator, or you can optionally add its contents to a variable in the Plan and then use a script task to create the file and set its permissions before the git checkout

Notes

The issue is linked to a bug on Apache-SSHD 2.7.0 bundled with Bamboo. It is fixed on Apache-SSHD 2.8.0:

• Wrong server key algorithm choose