Uploaded image for project: 'Bamboo Data Center'
  1. Bamboo Data Center
  2. BAM-21279

Increase Bamboo SSH Server timeout

XMLWordPrintable

    • Our product teams collect and evaluate feedback from a number of different sources. To learn more about how we use customer feedback in the planning process, check out our new feature policy.

      Problem Definition

      If the Git server is spending a long time packing objects, Bamboo's SSH Server (for the SSH proxying) can timeout after 10 mins 15 seconds of no read activity with InterruptedByTimeoutException and Git reports fatal: the remote end hung up unexpectedly fatal: protocol error: bad pack header. There is no way to extend this timeout.

      This will cause Bamboo to delete the Git cache and if there was 10 mins and 15 secs of packing activity, chances are its a very large repository. Getting a successful fetch after this can be challenging without being able to extend the timeout.

      2021-03-23 15:22:52,403 DEBUG [sshd-SshServer[1cf5234]-nio2-thread-5] [Nio2Session] handleReadCycleCompletion(Nio2Session[local=/127.0.0.1:46453, remote=/127.0.0.1:52968]) read 352 bytes
..
2021-03-23 15:33:07,791 WARN [sshd-SshServer[1cf5234]-nio2-thread-5] [ServerSessionImpl] exceptionCaught(ServerSessionImpl[f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@/127.0.0.1:52972])[state=Opened] InterruptedByTimeoutException: null
2021-03-23 15:33:07,791 DEBUG [sshd-SshServer[1cf5234]-nio2-thread-5] [ServerSessionImpl] exceptionCaught(ServerSessionImpl[f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@/127.0.0.1:52972])[state=Opened] details
java.nio.channels.InterruptedByTimeoutException
	at sun.nio.ch.UnixAsynchronousSocketChannelImpl$1.run(UnixAsynchronousSocketChannelImpl.java:458)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-03-23 15:33:07,808 WARN [10-BAM::PlanExec:pool-16-thread-3] [AbstractGitExecutor] Deleting git workspace /var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038225a4c969dfd1239b55c11fcdd848 due to exception
com.atlassian.bamboo.repository.RepositoryException: : Cannot fetch branch 'refs/heads/master' from 'ssh://git@bitbucket:7999/mir/git-hw.git' to source directory '/var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848'. command [/usr/bin/git fetch ssh://f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@127.0.0.1:46453/mir/git-hw.git +refs/heads/master:refs/heads/master --update-head-ok] failed with code 128. Working directory was [/var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848]., stderr:
Warning: Permanently added '[127.0.0.1]:46453' (RSA) to the list of known hosts.
Connection to 127.0.0.1 closed by remote host.
fatal: the remote end hung up unexpectedly
fatal: protocol error: bad pack header
	at com.atlassian.bamboo.plugins.git.NativeGitOperationHelper.fetch(NativeGitOperationHelper.java:545)
	at com.atlassian.bamboo.plugins.git.NativeGitOperationHelper.fetch(NativeGitOperationHelper.java:514)
	at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.ensureRevisionExists(GitChangeDetector.java:261)
	at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.lambda$fetchAndExtractChanges$4(GitChangeDetector.java:228)
	at io.atlassian.util.concurrent.ManagedLocks$ManagedLockImpl.withLock(ManagedLocks.java:284)
	at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.fetchAndExtractChanges(GitChangeDetector.java:226)
	at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.collectChanges(GitChangeDetector.java:196)

      Suggested Solution

      Provide an argument to extend Bamboo SSH Server timeouts.

      Workaround

      Anything that speeds up compression on the Git server end so that it completes before the Bamboo timeout is helpful. For example, if you binary files that aren't tracked by LFS you exclude them from delta compression using the .gitattributes file:

          Form Name

            [BAM-21279] Increase Bamboo SSH Server timeout

            Vani added a comment -

            There're few new system properties available at Bamboo Server and remote agent since v8.0

            bamboo.ssh.proxy.server.read.timeout - read timeout for Ssh Proxy Server which might be useful for large busy repositories checkout when packing of repo at Git side takes more than 10 minutes
bamboo.ssh.proxy.server.idle.timeout - idle timeout configuration for Ssh Proxy Server
bamboo.ssh.proxy.client.idle.timeout - Ssh Proxy client read timeout might be useful when Server timeout settings were increased
bamboo.ssh.proxy.client.auth.timeout - Ssh Proxy client auth timeout

            Vani added a comment - There're few new system properties available at Bamboo Server and remote agent since v8.0 bamboo.ssh.proxy.server.read.timeout - read timeout for Ssh Proxy Server which might be useful for large busy repositories checkout when packing of repo at Git side takes more than 10 minutes bamboo.ssh.proxy.server.idle.timeout - idle timeout configuration for Ssh Proxy Server bamboo.ssh.proxy.client.idle.timeout - Ssh Proxy client read timeout might be useful when Server timeout settings were increased bamboo.ssh.proxy.client.auth.timeout - Ssh Proxy client auth timeout

              achystoprudov Alexey Chystoprudov
              jowen@atlassian.com Jeremy Owen
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: