Details
-
Suggestion
-
Resolution: Fixed
-
None
Description
Problem Definition
If the Git server is spending a long time packing objects, Bamboo's SSH Server (for the SSH proxying) can timeout after 10 mins 15 seconds of no read activity with InterruptedByTimeoutException and Git reports fatal: the remote end hung up unexpectedly fatal: protocol error: bad pack header. There is no way to extend this timeout.
This will cause Bamboo to delete the Git cache and if there was 10 mins and 15 secs of packing activity, chances are its a very large repository. Getting a successful fetch after this can be challenging without being able to extend the timeout.
2021-03-23 15:22:52,403 DEBUG [sshd-SshServer[1cf5234]-nio2-thread-5] [Nio2Session] handleReadCycleCompletion(Nio2Session[local=/127.0.0.1:46453, remote=/127.0.0.1:52968]) read 352 bytes .. 2021-03-23 15:33:07,791 WARN [sshd-SshServer[1cf5234]-nio2-thread-5] [ServerSessionImpl] exceptionCaught(ServerSessionImpl[f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@/127.0.0.1:52972])[state=Opened] InterruptedByTimeoutException: null 2021-03-23 15:33:07,791 DEBUG [sshd-SshServer[1cf5234]-nio2-thread-5] [ServerSessionImpl] exceptionCaught(ServerSessionImpl[f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@/127.0.0.1:52972])[state=Opened] details java.nio.channels.InterruptedByTimeoutException at sun.nio.ch.UnixAsynchronousSocketChannelImpl$1.run(UnixAsynchronousSocketChannelImpl.java:458) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) 2021-03-23 15:33:07,808 WARN [10-BAM::PlanExec:pool-16-thread-3] [AbstractGitExecutor] Deleting git workspace /var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038225a4c969dfd1239b55c11fcdd848 due to exception com.atlassian.bamboo.repository.RepositoryException: : Cannot fetch branch 'refs/heads/master' from 'ssh://git@bitbucket:7999/mir/git-hw.git' to source directory '/var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848'. command [/usr/bin/git fetch ssh://f7a9f8fb-1d34-4c74-b092-a2cd8bb1c2c8@127.0.0.1:46453/mir/git-hw.git +refs/heads/master:refs/heads/master --update-head-ok] failed with code 128. Working directory was [/var/bamboo-home/main/xml-data/build-dir/_git-repositories-cache/7770976a038865a4c969dfd1239b55c11fcdd848]., stderr: Warning: Permanently added '[127.0.0.1]:46453' (RSA) to the list of known hosts. Connection to 127.0.0.1 closed by remote host. fatal: the remote end hung up unexpectedly fatal: protocol error: bad pack header at com.atlassian.bamboo.plugins.git.NativeGitOperationHelper.fetch(NativeGitOperationHelper.java:545) at com.atlassian.bamboo.plugins.git.NativeGitOperationHelper.fetch(NativeGitOperationHelper.java:514) at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.ensureRevisionExists(GitChangeDetector.java:261) at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.lambda$fetchAndExtractChanges$4(GitChangeDetector.java:228) at io.atlassian.util.concurrent.ManagedLocks$ManagedLockImpl.withLock(ManagedLocks.java:284) at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.fetchAndExtractChanges(GitChangeDetector.java:226) at com.atlassian.bamboo.plugins.git.v2.GitChangeDetector.collectChanges(GitChangeDetector.java:196)
Suggested Solution
Provide an argument to extend Bamboo SSH Server timeouts.
Workaround
Anything that speeds up compression on the Git server end so that it completes before the Bamboo timeout is helpful. For example, if you binary files that aren't tracked by LFS you exclude them from delta compression using the .gitattributes file:
Attachments
Issue Links
- relates to
-
- Closed