Problem

Git checkout tasks are extremely slow when done via the Source Code Checkout task through BAMBOO-SSH-PROXY

Environment

Bamboo 8.0.4 or later (apache-sshd-2.7.0+). Earlier releases which use apache-sshd-1.7.0 are not prone to this issue.

Steps to Reproduce

Add a large repository to your Linked Repositories in Bamboo (4GB+, mysql-server for example)

Create a code checkout task in a Plan

Run the plan and observe the 'git fetch' low throughput.

Expected Results

The git fetch command throughput via BAMBOO-SSH-PROXY should be equivalent or really close the similar git command when reaching the target repository directly without a proxy.

Actual Results

Git checkout via the BAMBOO-SSH-PROXY from a remote agent or local agent is extremely slow (up to 2 or 3Mb/s maximum). Taking a considerable amount of time to complete any checkout task.

simple  18-Dec-2021 05:08:16    Receiving objects:  38% (49055/127204), 279.71 MiB | 1.21 MiB/s
simple  18-Dec-2021 05:08:17    Receiving objects:  39% (49610/127204), 280.36 MiB | 1.21 MiB/s
simple  18-Dec-2021 05:08:17    Receiving objects:  39% (50100/127204), 281.02 MiB | 1.22 MiB/s
simple  18-Dec-2021 05:08:18    Receiving objects:  39% (50730/127204), 282.33 MiB | 1.21 MiB/s

Workaround

Enable Repository caching on agents and possibly Shallow clones.

• Bamboo Administration -> Linked Repositories -> Select the repository -> Advanced Options -> Enable repository caching on Agents

You can also enable Use shallow clones if your builds don't depend on the repository history. That option is right below "Enable repository caching".

Enabling the two options above will speed up things considerably from the second time the build runs on the same agent. The agent needs to check out the full repository at least once so it can have a full copy of it.

Note: If applying this workaround, make sure not to delete any of the relevant git repository caches that are stored locally on the agent, otherwise, a full checkout will be performed again via the BAMBOO-SSH-PROXY (slower).

And/or:

Checkout source code as a Script task

You can optionally check out the source code as a regular "git" command within a Script task. This will bypass BAMBOO-SSH-PROXY and speed up things. You will need to configure git authentication manually on each agent.

#!/bin/bash -x
# Your remote/local agent must have proper SSH keys exchanged with your Git repository
# This script will not manage authentication

# This code is untested and may need further adjustments to suit your requirements
# Use this script only if after finding that enabling Repository Caching on your repositories 
# does not suit you
REPOSITORY=${bamboo.planRepository.repositoryUrl}
REVISION=${bamboo.planRepository.revision}
BRANCHNAME=${bamboo.planRepository.branchName}
GIT=${bamboo.capability.system.git.executable}
GIT_SSH_COMMAND="ssh -o StrictHostKeyChecking=no -o BatchMode=yes -o UserKnownHostsFile=/dev/null"

${GIT} init
${GIT} config remote.origin.url >&- || ${GIT} remote add origin ${REPOSITORY}

# [0] = REVISION
# [1] = Branch/HEAD
CUSTOM_GIT_HEADS=($(${GIT} ls-remote --heads ${REPOSITORY}))

${GIT} fetch ${REPOSITORY} +${CUSTOM_GIT_HEADS[1]}:${CUSTOM_GIT_HEADS[1]} --update-head-ok --progress --verbose
${GIT} show-ref ${BRANCHNAME}

# When specific revision is requested via custom build or rerun
if [ ${CUSTOM_GIT_HEADS[0]} != ${REVISION} ] ; then
  ${GIT} checkout -f ${REVISION}
else
  ${GIT} checkout -f ${BRANCHNAME}
fi