Uploaded image for project: 'Bamboo Data Center'
  1. Bamboo Data Center
  2. BAM-21279

Increase Bamboo SSH Server timeout

XMLWordPrintable

    • Our product teams collect and evaluate feedback from a number of different sources. To learn more about how we use customer feedback in the planning process, check out our new feature policy.

      Problem Definition

      If the Git server is spending a long time packing objects, Bamboo's SSH Server (for the SSH proxying) can timeout after 10 mins 15 seconds of no read activity with InterruptedByTimeoutException and Git reports fatal: the remote end hung up unexpectedly fatal: protocol error: bad pack header. There is no way to extend this timeout.

      This will cause Bamboo to delete the Git cache and if there was 10 mins and 15 secs of packing activity, chances are its a very large repository. Getting a successful fetch after this can be challenging without being able to extend the timeout.

      2021-03-23 15:22:52,403 DEBUG [sshd-SshServer[1cf5234]-nio2-thread-5] [Nio2Session] handleReadCycleCompletion(Nio2Session[local=/127.0.0.1:46453, remote=/127.0.0.1:52968]) read 352 bytes
..
2021-03-23 15:33:07,791 WARN [sshd-SshServer[1cf5234]-nio2-thread-5] [ServerSessionImpl] exceptionCaught(ServerSessionImpl[f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@/127.0.0.1:52972])[state=Opened] InterruptedByTimeoutException: null
2021-03-23 15:33:07,791 DEBUG [sshd-SshServer[1cf5234]-nio2-thread-5] [ServerSessionImpl] exceptionCaught(ServerSessionImpl[f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@/127.0.0.1:52972])[state=Opened] details
java.nio.channels.InterruptedByTimeoutException
	at sun.nio.ch.UnixAsynchronousSocketChannelImpl$1.run(UnixAsynchronousSocketChannelImpl.java:458)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
2021-03-23 15:33:07,808 WARN [10-BAM::PlanExec:pool-16-thread-3] [AbstractGitExecutor] Deleting git workspace /var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038225a4c969dfd1239b55c11fcdd848 due to exception
com.atlassian.bamboo.repository.RepositoryException: : Cannot fetch branch 'refs/heads/master' from 'ssh://git@bitbucket:7999/mir/git-hw.git' to source directory '/var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848'. command [/usr/bin/git fetch ssh://f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@127.0.0.1:46453/mir/git-hw.git +refs/heads/master:refs/heads/master --update-head-ok] failed with code 128. Working directory was [/var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848]., stderr:
Warning: Permanently added '[127.0.0.1]:46453' (RSA) to the list of known hosts.
Connection to 127.0.0.1 closed by remote host.
fatal: the remote end hung up unexpectedly
fatal: protocol error: bad pack header
	at com.atlassian.bamboo.plugins.git.NativeGitOperationHelper.fetch(NativeGitOperationHelper.java:545)
	at com.atlassian.bamboo.plugins.git.NativeGitOperationHelper.fetch(NativeGitOperationHelper.java:514)
	at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.ensureRevisionExists(GitChangeDetector.java:261)
	at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.lambda$fetchAndExtractChanges$4(GitChangeDetector.java:228)
	at io.atlassian.util.concurrent.ManagedLocks$ManagedLockImpl.withLock(ManagedLocks.java:284)
	at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.fetchAndExtractChanges(GitChangeDetector.java:226)
	at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.collectChanges(GitChangeDetector.java:196)

      Suggested Solution

      Provide an argument to extend Bamboo SSH Server timeouts.

      Workaround

      Anything that speeds up compression on the Git server end so that it completes before the Bamboo timeout is helpful. For example, if you binary files that aren't tracked by LFS you exclude them from delta compression using the .gitattributes file:

          Form Name

            [BAM-21279] Increase Bamboo SSH Server timeout

            Bruno Rosa made changes -
            Remote Link New: This issue links to "Page (Confluence)" [ 946946 ]
            Marcin Gardias made changes -
            Resolution New: Fixed [ 1 ]
            Status Original: Waiting for Release [ 12075 ] New: Closed [ 6 ]
            Alexey Chystoprudov made changes -
            Fix Version/s New: 8.0.0 EAP1 [ 95495 ]
            Jeremy Owen made changes -
            Description Original: h2. Problem Definition

            If the Git server is spending a long time packing objects, Bamboo's SSH Server (for the SSH proxying) can timeout after 10 mins 15 seconds of no read activity with {{InterruptedByTimeoutException}} and Git reports {{fatal: the remote end hung up unexpectedly fatal: protocol error: bad pack header}}. There is no way to extend this timeout.

            This will cause Bamboo to delete the Git cache and if there was 10 mins and 15 secs of packing activity, chances are its a very large repository. Getting a successful fetch after this can be challenging without being able to extend the timeout.

            {noformat}
            2021-03-23 15:22:52,403 DEBUG [sshd-SshServer[1cf5234]-nio2-thread-5] [Nio2Session] handleReadCycleCompletion(Nio2Session[local=/127.0.0.1:46453, remote=/127.0.0.1:52968]) read 352 bytes
            ..
            2021-03-23 15:33:07,791 WARN [sshd-SshServer[1cf5234]-nio2-thread-5] [ServerSessionImpl] exceptionCaught(ServerSessionImpl[f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@/127.0.0.1:52972])[state=Opened] InterruptedByTimeoutException: null
            2021-03-23 15:33:07,791 DEBUG [sshd-SshServer[1cf5234]-nio2-thread-5] [ServerSessionImpl] exceptionCaught(ServerSessionImpl[f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@/127.0.0.1:52972])[state=Opened] details
            java.nio.channels.InterruptedByTimeoutException
            at sun.nio.ch.UnixAsynchronousSocketChannelImpl$1.run(UnixAsynchronousSocketChannelImpl.java:458)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
            2021-03-23 15:33:07,808 WARN [10-BAM::PlanExec:pool-16-thread-3] [AbstractGitExecutor] Deleting git workspace /var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848 due to exception
            com.atlassian.bamboo.repository.RepositoryException: : Cannot fetch branch 'refs/heads/master' from 'ssh://git@bitbucket:7999/mir/git-hw.git' to source directory '/var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848'. command [/usr/bin/git fetch ssh://f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@127.0.0.1:46453/mir/git-hw.git +refs/heads/master:refs/heads/master --update-head-ok] failed with code 128. Working directory was [/var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848]., stderr:
            Warning: Permanently added '[127.0.0.1]:46453' (RSA) to the list of known hosts.
            Connection to 127.0.0.1 closed by remote host.
            fatal: the remote end hung up unexpectedly
            fatal: protocol error: bad pack header
            at com.atlassian.bamboo.plugins.git.NativeGitOperationHelper.fetch(NativeGitOperationHelper.java:545)
            at com.atlassian.bamboo.plugins.git.NativeGitOperationHelper.fetch(NativeGitOperationHelper.java:514)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.ensureRevisionExists(GitChangeDetector.java:261)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.lambda$fetchAndExtractChanges$4(GitChangeDetector.java:228)
            at io.atlassian.util.concurrent.ManagedLocks$ManagedLockImpl.withLock(ManagedLocks.java:284)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.fetchAndExtractChanges(GitChangeDetector.java:226)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.collectChanges(GitChangeDetector.java:196)
            {noformat}

            h2. Suggested Solution

            Provide an argument to extend Bamboo SSH Server timeouts.

            h2. Workaround
            Anything that speeds up compression on the Git server end so that it completes before the Bamboo timeout is helpful. For example, if you binary files that aren't tracked by LFS you exclude them from delta compression using the {{.gitattributes}} file:
            * [https://git-scm.com/docs/gitattributes|https://git-scm.com/docs/gitattributes]
            		New: h2. Problem Definition

            If the Git server is spending a long time packing objects, Bamboo's SSH Server (for the SSH proxying) can timeout after 10 mins 15 seconds of no read activity with {{InterruptedByTimeoutException}} and Git reports {{fatal: the remote end hung up unexpectedly fatal: protocol error: bad pack header}}. There is no way to extend this timeout.

            This will cause Bamboo to delete the Git cache and if there was 10 mins and 15 secs of packing activity, chances are its a very large repository. Getting a successful fetch after this can be challenging without being able to extend the timeout.

            {noformat}
            2021-03-23 15:22:52,403 DEBUG [sshd-SshServer[1cf5234]-nio2-thread-5] [Nio2Session] handleReadCycleCompletion(Nio2Session[local=/127.0.0.1:46453, remote=/127.0.0.1:52968]) read 352 bytes
            ..
            2021-03-23 15:33:07,791 WARN [sshd-SshServer[1cf5234]-nio2-thread-5] [ServerSessionImpl] exceptionCaught(ServerSessionImpl[f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@/127.0.0.1:52972])[state=Opened] InterruptedByTimeoutException: null
            2021-03-23 15:33:07,791 DEBUG [sshd-SshServer[1cf5234]-nio2-thread-5] [ServerSessionImpl] exceptionCaught(ServerSessionImpl[f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@/127.0.0.1:52972])[state=Opened] details
            java.nio.channels.InterruptedByTimeoutException
            at sun.nio.ch.UnixAsynchronousSocketChannelImpl$1.run(UnixAsynchronousSocketChannelImpl.java:458)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
            2021-03-23 15:33:07,808 WARN [10-BAM::PlanExec:pool-16-thread-3] [AbstractGitExecutor] Deleting git workspace /var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038225a4c969dfd1239b55c11fcdd848 due to exception
            com.atlassian.bamboo.repository.RepositoryException: : Cannot fetch branch 'refs/heads/master' from 'ssh://git@bitbucket:7999/mir/git-hw.git' to source directory '/var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848'. command [/usr/bin/git fetch ssh://f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@127.0.0.1:46453/mir/git-hw.git +refs/heads/master:refs/heads/master --update-head-ok] failed with code 128. Working directory was [/var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848]., stderr:
            Warning: Permanently added '[127.0.0.1]:46453' (RSA) to the list of known hosts.
            Connection to 127.0.0.1 closed by remote host.
            fatal: the remote end hung up unexpectedly
            fatal: protocol error: bad pack header
            at com.atlassian.bamboo.plugins.git.NativeGitOperationHelper.fetch(NativeGitOperationHelper.java:545)
            at com.atlassian.bamboo.plugins.git.NativeGitOperationHelper.fetch(NativeGitOperationHelper.java:514)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.ensureRevisionExists(GitChangeDetector.java:261)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.lambda$fetchAndExtractChanges$4(GitChangeDetector.java:228)
            at io.atlassian.util.concurrent.ManagedLocks$ManagedLockImpl.withLock(ManagedLocks.java:284)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.fetchAndExtractChanges(GitChangeDetector.java:226)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.collectChanges(GitChangeDetector.java:196)
            {noformat}

            h2. Suggested Solution

            Provide an argument to extend Bamboo SSH Server timeouts.

            h2. Workaround
            Anything that speeds up compression on the Git server end so that it completes before the Bamboo timeout is helpful. For example, if you binary files that aren't tracked by LFS you exclude them from delta compression using the {{.gitattributes}} file:
            * [https://git-scm.com/docs/gitattributes|https://git-scm.com/docs/gitattributes]
            Jeremy Owen made changes -
            Fix Version/s New: 8.0.0 [ 92814 ]
            Jeremy Owen made changes -
            Status Original: In Progress [ 3 ] New: Waiting for Release [ 12075 ]
            Jeremy Owen made changes -
            Status Original: Gathering Interest [ 11772 ] New: In Progress [ 3 ]
            Jeremy Owen made changes -
            Assignee New: Alexey Chystoprudov [ achystoprudov ]
            Jeremy Owen made changes -
            Description Original: h2. Problem Definition

            If the Git server is spending a long time packing objects, Bamboo's SSH Server (for the SSH proxying) can timeout after 10 mins 15 seconds of no read activity with {{InterruptedByTimeoutException}} and Git reports {{fatal: the remote end hung up unexpectedly fatal: protocol error: bad pack header}}. There is no way to extend this timeout.

            This will cause Bamboo to delete the Git cache and if there was 10 mins and 15 secs of packing activity, chances are its a very large repository. Getting a successful fetch after this can be challenging without being able to extend the timeout.

            {noformat}
            2021-03-23 15:22:52,403 DEBUG [sshd-SshServer[1cf5234]-nio2-thread-5] [Nio2Session] handleReadCycleCompletion(Nio2Session[local=/127.0.0.1:46453, remote=/127.0.0.1:52968]) read 352 bytes
            ..
            2021-03-23 15:33:07,791 WARN [sshd-SshServer[1cf5234]-nio2-thread-5] [ServerSessionImpl] exceptionCaught(ServerSessionImpl[f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@/127.0.0.1:52972])[state=Opened] InterruptedByTimeoutException: null
            2021-03-23 15:33:07,791 DEBUG [sshd-SshServer[1cf5234]-nio2-thread-5] [ServerSessionImpl] exceptionCaught(ServerSessionImpl[f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@/127.0.0.1:52972])[state=Opened] details
            java.nio.channels.InterruptedByTimeoutException
            at sun.nio.ch.UnixAsynchronousSocketChannelImpl$1.run(UnixAsynchronousSocketChannelImpl.java:458)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
            2021-03-23 15:33:07,808 WARN [10-BAM::PlanExec:pool-16-thread-3] [AbstractGitExecutor] Deleting git workspace /var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848 due to exception
            com.atlassian.bamboo.repository.RepositoryException: : Cannot fetch branch 'refs/heads/master' from 'ssh://git@bitbucket:7999/mir/git-hw.git' to source directory '/var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848'. command [/usr/bin/git fetch ssh://f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@127.0.0.1:46453/mir/git-hw.git +refs/heads/master:refs/heads/master --update-head-ok] failed with code 128. Working directory was [/var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848]., stderr:
            Warning: Permanently added '[127.0.0.1]:46453' (RSA) to the list of known hosts.
            Connection to 127.0.0.1 closed by remote host.
            fatal: the remote end hung up unexpectedly
            fatal: protocol error: bad pack header
            at com.atlassian.bamboo.plugins.git.NativeGitOperationHelper.fetch(NativeGitOperationHelper.java:545)
            at com.atlassian.bamboo.plugins.git.NativeGitOperationHelper.fetch(NativeGitOperationHelper.java:514)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.ensureRevisionExists(GitChangeDetector.java:261)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.lambda$fetchAndExtractChanges$4(GitChangeDetector.java:228)
            at io.atlassian.util.concurrent.ManagedLocks$ManagedLockImpl.withLock(ManagedLocks.java:284)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.fetchAndExtractChanges(GitChangeDetector.java:226)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.collectChanges(GitChangeDetector.java:196)
            {noformat}

            h2. Suggested Solution

            Provide an argument to extend Bamboo SSH Server timeouts.

            h2. Workaround
            Anything that speeds up compression on the Git server end is helpful. For example, if you binary files that aren't tracked by LFS you exclude them from delta compression using the {{.gitattributes}} file:
            * [https://git-scm.com/docs/gitattributes|https://git-scm.com/docs/gitattributes]
            		New: h2. Problem Definition

            If the Git server is spending a long time packing objects, Bamboo's SSH Server (for the SSH proxying) can timeout after 10 mins 15 seconds of no read activity with {{InterruptedByTimeoutException}} and Git reports {{fatal: the remote end hung up unexpectedly fatal: protocol error: bad pack header}}. There is no way to extend this timeout.

            This will cause Bamboo to delete the Git cache and if there was 10 mins and 15 secs of packing activity, chances are its a very large repository. Getting a successful fetch after this can be challenging without being able to extend the timeout.

            {noformat}
            2021-03-23 15:22:52,403 DEBUG [sshd-SshServer[1cf5234]-nio2-thread-5] [Nio2Session] handleReadCycleCompletion(Nio2Session[local=/127.0.0.1:46453, remote=/127.0.0.1:52968]) read 352 bytes
            ..
            2021-03-23 15:33:07,791 WARN [sshd-SshServer[1cf5234]-nio2-thread-5] [ServerSessionImpl] exceptionCaught(ServerSessionImpl[f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@/127.0.0.1:52972])[state=Opened] InterruptedByTimeoutException: null
            2021-03-23 15:33:07,791 DEBUG [sshd-SshServer[1cf5234]-nio2-thread-5] [ServerSessionImpl] exceptionCaught(ServerSessionImpl[f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@/127.0.0.1:52972])[state=Opened] details
            java.nio.channels.InterruptedByTimeoutException
            at sun.nio.ch.UnixAsynchronousSocketChannelImpl$1.run(UnixAsynchronousSocketChannelImpl.java:458)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
            2021-03-23 15:33:07,808 WARN [10-BAM::PlanExec:pool-16-thread-3] [AbstractGitExecutor] Deleting git workspace /var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848 due to exception
            com.atlassian.bamboo.repository.RepositoryException: : Cannot fetch branch 'refs/heads/master' from 'ssh://git@bitbucket:7999/mir/git-hw.git' to source directory '/var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848'. command [/usr/bin/git fetch ssh://f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@127.0.0.1:46453/mir/git-hw.git +refs/heads/master:refs/heads/master --update-head-ok] failed with code 128. Working directory was [/var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848]., stderr:
            Warning: Permanently added '[127.0.0.1]:46453' (RSA) to the list of known hosts.
            Connection to 127.0.0.1 closed by remote host.
            fatal: the remote end hung up unexpectedly
            fatal: protocol error: bad pack header
            at com.atlassian.bamboo.plugins.git.NativeGitOperationHelper.fetch(NativeGitOperationHelper.java:545)
            at com.atlassian.bamboo.plugins.git.NativeGitOperationHelper.fetch(NativeGitOperationHelper.java:514)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.ensureRevisionExists(GitChangeDetector.java:261)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.lambda$fetchAndExtractChanges$4(GitChangeDetector.java:228)
            at io.atlassian.util.concurrent.ManagedLocks$ManagedLockImpl.withLock(ManagedLocks.java:284)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.fetchAndExtractChanges(GitChangeDetector.java:226)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.collectChanges(GitChangeDetector.java:196)
            {noformat}

            h2. Suggested Solution

            Provide an argument to extend Bamboo SSH Server timeouts.

            h2. Workaround
            Anything that speeds up compression on the Git server end so that it completes before the Bamboo timeout is helpful. For example, if you binary files that aren't tracked by LFS you exclude them from delta compression using the {{.gitattributes}} file:
            * [https://git-scm.com/docs/gitattributes|https://git-scm.com/docs/gitattributes]
            Jeremy Owen made changes -
            Description Original: h2. Problem Definition

            If the Git server is spending a long time packing objects, Bamboo's SSH Server (for the SSH proxying) can timeout after 10 mins 15 seconds of no read activity with {{InterruptedByTimeoutException}} and Git reports {{fatal: the remote end hung up unexpectedly fatal: protocol error: bad pack header}}. There is no way to extend this timeout.

            This will cause Bamboo to delete the Git cache and if there was 10 mins and 15 secs of packing activity, chances are its a very large repository. Getting a successful fetch after this can be challenging without being able to extend the timeout.

            {noformat}
            2021-03-23 15:22:52,403 DEBUG [sshd-SshServer[1cf5234]-nio2-thread-5] [Nio2Session] handleReadCycleCompletion(Nio2Session[local=/127.0.0.1:46453, remote=/127.0.0.1:52968]) read 352 bytes
            ..
            2021-03-23 15:33:07,791 WARN [sshd-SshServer[1cf5234]-nio2-thread-5] [ServerSessionImpl] exceptionCaught(ServerSessionImpl[f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@/127.0.0.1:52972])[state=Opened] InterruptedByTimeoutException: null
            2021-03-23 15:33:07,791 DEBUG [sshd-SshServer[1cf5234]-nio2-thread-5] [ServerSessionImpl] exceptionCaught(ServerSessionImpl[f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@/127.0.0.1:52972])[state=Opened] details
            java.nio.channels.InterruptedByTimeoutException
            at sun.nio.ch.UnixAsynchronousSocketChannelImpl$1.run(UnixAsynchronousSocketChannelImpl.java:458)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
            2021-03-23 15:33:07,808 WARN [10-BAM::PlanExec:pool-16-thread-3] [AbstractGitExecutor] Deleting git workspace /var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848 due to exception
            com.atlassian.bamboo.repository.RepositoryException: : Cannot fetch branch 'refs/heads/master' from 'ssh://git@bitbucket:7999/mir/git-hw.git' to source directory '/var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848'. command [/usr/bin/git fetch ssh://f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@127.0.0.1:46453/mir/git-hw.git +refs/heads/master:refs/heads/master --update-head-ok] failed with code 128. Working directory was [/var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848]., stderr:
            Warning: Permanently added '[127.0.0.1]:46453' (RSA) to the list of known hosts.
            Connection to 127.0.0.1 closed by remote host.
            fatal: the remote end hung up unexpectedly
            fatal: protocol error: bad pack header
            at com.atlassian.bamboo.plugins.git.NativeGitOperationHelper.fetch(NativeGitOperationHelper.java:545)
            at com.atlassian.bamboo.plugins.git.NativeGitOperationHelper.fetch(NativeGitOperationHelper.java:514)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.ensureRevisionExists(GitChangeDetector.java:261)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.lambda$fetchAndExtractChanges$4(GitChangeDetector.java:228)
            at io.atlassian.util.concurrent.ManagedLocks$ManagedLockImpl.withLock(ManagedLocks.java:284)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.fetchAndExtractChanges(GitChangeDetector.java:226)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.collectChanges(GitChangeDetector.java:196)
            {noformat}

            h2. Suggested Solution

            Provide an argument to extend Bamboo SSH Server timeouts.             		New: h2. Problem Definition

            If the Git server is spending a long time packing objects, Bamboo's SSH Server (for the SSH proxying) can timeout after 10 mins 15 seconds of no read activity with {{InterruptedByTimeoutException}} and Git reports {{fatal: the remote end hung up unexpectedly fatal: protocol error: bad pack header}}. There is no way to extend this timeout.

            This will cause Bamboo to delete the Git cache and if there was 10 mins and 15 secs of packing activity, chances are its a very large repository. Getting a successful fetch after this can be challenging without being able to extend the timeout.

            {noformat}
            2021-03-23 15:22:52,403 DEBUG [sshd-SshServer[1cf5234]-nio2-thread-5] [Nio2Session] handleReadCycleCompletion(Nio2Session[local=/127.0.0.1:46453, remote=/127.0.0.1:52968]) read 352 bytes
            ..
            2021-03-23 15:33:07,791 WARN [sshd-SshServer[1cf5234]-nio2-thread-5] [ServerSessionImpl] exceptionCaught(ServerSessionImpl[f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@/127.0.0.1:52972])[state=Opened] InterruptedByTimeoutException: null
            2021-03-23 15:33:07,791 DEBUG [sshd-SshServer[1cf5234]-nio2-thread-5] [ServerSessionImpl] exceptionCaught(ServerSessionImpl[f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@/127.0.0.1:52972])[state=Opened] details
            java.nio.channels.InterruptedByTimeoutException
            at sun.nio.ch.UnixAsynchronousSocketChannelImpl$1.run(UnixAsynchronousSocketChannelImpl.java:458)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
            2021-03-23 15:33:07,808 WARN [10-BAM::PlanExec:pool-16-thread-3] [AbstractGitExecutor] Deleting git workspace /var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848 due to exception
            com.atlassian.bamboo.repository.RepositoryException: : Cannot fetch branch 'refs/heads/master' from 'ssh://git@bitbucket:7999/mir/git-hw.git' to source directory '/var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848'. command [/usr/bin/git fetch ssh://f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@127.0.0.1:46453/mir/git-hw.git +refs/heads/master:refs/heads/master --update-head-ok] failed with code 128. Working directory was [/var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848]., stderr:
            Warning: Permanently added '[127.0.0.1]:46453' (RSA) to the list of known hosts.
            Connection to 127.0.0.1 closed by remote host.
            fatal: the remote end hung up unexpectedly
            fatal: protocol error: bad pack header
            at com.atlassian.bamboo.plugins.git.NativeGitOperationHelper.fetch(NativeGitOperationHelper.java:545)
            at com.atlassian.bamboo.plugins.git.NativeGitOperationHelper.fetch(NativeGitOperationHelper.java:514)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.ensureRevisionExists(GitChangeDetector.java:261)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.lambda$fetchAndExtractChanges$4(GitChangeDetector.java:228)
            at io.atlassian.util.concurrent.ManagedLocks$ManagedLockImpl.withLock(ManagedLocks.java:284)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.fetchAndExtractChanges(GitChangeDetector.java:226)
            at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.collectChanges(GitChangeDetector.java:196)
            {noformat}

            h2. Suggested Solution

            Provide an argument to extend Bamboo SSH Server timeouts.

            h2. Workaround
            Anything that speeds up compression on the Git server end is helpful. For example, if you binary files that aren't tracked by LFS you exclude them from delta compression using the {{.gitattributes}} file:
            * [https://git-scm.com/docs/gitattributes|https://git-scm.com/docs/gitattributes]

              achystoprudov Alexey Chystoprudov
              jowen@atlassian.com Jeremy Owen
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: