BSERV-10100 introduced throttling of SSH writes when the remote SSH client specifies a large SSH window size. Recent versions of PuTTY request a windows size of 2GB and the throttling was necessary to prevent OutOfMemoryErrors.
Unfortunately, the fix for BSERV-10100 introduced a race condition which can result in a deadlock when an SSH connection is terminated by the remote client while the pending writes to the client are throttled. This deadlock affects the SSH I/O processing thread(s). SSH sessions are mapped to I/O processing threads. Any (new) sessions mapped to a deadlocked I/O processing thread will hang without any I/O processing.
Symptoms
Some git operations over SSH will hang without any output and fail after a timeout occurs.
The atlassian-bitbucket.log contains "Too many open files" errors:
java.io.IOException: Too many open files
Diagnosis
Take a thread dump of the Bitbucket Server instance and look for threads with a name starting with NioProcessor-. If these threads have stacktrace similar to the following, the thread is deadlocked.
- org.apache.sshd.common.channel.ChannelOutputStream.close() @bci=0, line=232 (Compiled frame) - org.apache.sshd.common.util.io.IoUtils.closeQuietly(java.io.Closeable[]) @bci=29, line=127 (Compiled frame) - org.apache.sshd.server.channel.ChannelSession.doCloseImmediately() @bci=155, line=209 (Compiled frame) - org.apache.sshd.common.util.closeable.AbstractCloseable.close(boolean) @bci=65, line=81 (Compiled frame) - org.apache.sshd.common.channel.AbstractChannel.close(boolean) @bci=37, line=448 (Compiled frame) - com.atlassian.bitbucket.internal.ssh.server.ScmHostingChannelSession.close(boolean) @bci=2, line=47 (Compiled frame) - org.apache.sshd.common.util.closeable.ParallelCloseable.doClose(boolean) @bci=117, line=65 (Compiled frame) - org.apache.sshd.common.util.closeable.SimpleCloseable.close(boolean) @bci=14, line=63 (Compiled frame) - org.apache.sshd.common.util.closeable.AbstractInnerCloseable.doCloseImmediately() @bci=5, line=47 (Compiled frame)
- is caused by
-
BSERV-10100 OutOfMemoryError in NioSocketSession's DefaultWriteRequestQueue
-
- Closed
-
- causes
-
PS-22298 You do not have permission to view this issue
- is related to
-
SSP-28756 You do not have permission to view this issue
Hi all,
The custom SSH plugin that was attached has been removed from this ticket as we've just released the new official versions of Bitbucket that include this fix. All the fix versions listed in this issue are available for download and will ensure that you do not run into this issue.