Details
High Description
BSERV-10100 introduced throttling of SSH writes when the remote SSH client specifies a large SSH window size. Recent versions of PuTTY request a windows size of 2GB and the throttling was necessary to prevent OutOfMemoryErrors.
Unfortunately, the fix for BSERV-10100 introduced a race condition which can result in a deadlock when an SSH connection is terminated by the remote client while the pending writes to the client are throttled. This deadlock affects the SSH I/O processing thread(s). SSH sessions are mapped to I/O processing threads. Any (new) sessions mapped to a deadlocked I/O processing thread will hang without any I/O processing.
Symptoms
Some git operations over SSH will hang without any output and fail after a timeout occurs.
The atlassian-bitbucket.log contains "Too many open files" errors:
java.io.IOException: Too many open files
Diagnosis
Take a thread dump of the Bitbucket Server instance and look for threads with a name starting with NioProcessor-. If these threads have stacktrace similar to the following, the thread is deadlocked.
- org.apache.sshd.common.channel.ChannelOutputStream.close() @bci=0, line=232 (Compiled frame) - org.apache.sshd.common.util.io.IoUtils.closeQuietly(java.io.Closeable[]) @bci=29, line=127 (Compiled frame) - org.apache.sshd.server.channel.ChannelSession.doCloseImmediately() @bci=155, line=209 (Compiled frame) - org.apache.sshd.common.util.closeable.AbstractCloseable.close(boolean) @bci=65, line=81 (Compiled frame) - org.apache.sshd.common.channel.AbstractChannel.close(boolean) @bci=37, line=448 (Compiled frame) - com.atlassian.bitbucket.internal.ssh.server.ScmHostingChannelSession.close(boolean) @bci=2, line=47 (Compiled frame) - org.apache.sshd.common.util.closeable.ParallelCloseable.doClose(boolean) @bci=117, line=65 (Compiled frame) - org.apache.sshd.common.util.closeable.SimpleCloseable.close(boolean) @bci=14, line=63 (Compiled frame) - org.apache.sshd.common.util.closeable.AbstractInnerCloseable.doCloseImmediately() @bci=5, line=47 (Compiled frame)