Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-10575

Too many open files caused by a deadlock in SSH threads due to race condition with PuTTY SSH clients

XMLWordPrintable

      BSERV-10100 introduced throttling of SSH writes when the remote SSH client specifies a large SSH window size. Recent versions of PuTTY request a windows size of 2GB and the throttling was necessary to prevent OutOfMemoryErrors.

      Unfortunately, the fix for BSERV-10100 introduced a race condition which can result in a deadlock when an SSH connection is terminated by the remote client while the pending writes to the client are throttled. This deadlock affects the SSH I/O processing thread(s). SSH sessions are mapped to I/O processing threads. Any (new) sessions mapped to a deadlocked I/O processing thread will hang without any I/O processing.

      Symptoms
      Some git operations over SSH will hang without any output and fail after a timeout occurs.

      The atlassian-bitbucket.log contains "Too many open files" errors:

      java.io.IOException: Too many open files

      Diagnosis
      Take a thread dump of the Bitbucket Server instance and look for threads with a name starting with NioProcessor-. If these threads have stacktrace similar to the following, the thread is deadlocked. 

       - org.apache.sshd.common.channel.ChannelOutputStream.close() @bci=0, line=232 (Compiled frame)
 - org.apache.sshd.common.util.io.IoUtils.closeQuietly(java.io.Closeable[]) @bci=29, line=127 (Compiled frame)
 - org.apache.sshd.server.channel.ChannelSession.doCloseImmediately() @bci=155, line=209 (Compiled frame)
 - org.apache.sshd.common.util.closeable.AbstractCloseable.close(boolean) @bci=65, line=81 (Compiled frame)
 - org.apache.sshd.common.channel.AbstractChannel.close(boolean) @bci=37, line=448 (Compiled frame)
 - com.atlassian.bitbucket.internal.ssh.server.ScmHostingChannelSession.close(boolean) @bci=2, line=47 (Compiled frame)
 - org.apache.sshd.common.util.closeable.ParallelCloseable.doClose(boolean) @bci=117, line=65 (Compiled frame)
 - org.apache.sshd.common.util.closeable.SimpleCloseable.close(boolean) @bci=14, line=63 (Compiled frame)
 - org.apache.sshd.common.util.closeable.AbstractInnerCloseable.doCloseImmediately() @bci=5, line=47 (Compiled frame)

            [BSERV-10575] Too many open files caused by a deadlock in SSH threads due to race condition with PuTTY SSH clients

            No work has yet been logged on this issue.

              mheemskerk Michael Heemskerk (Inactive)
              mheemskerk Michael Heemskerk (Inactive)
              Affected customers:
              0 This affects my team
              Watchers:
              14 Start watching this issue

                Created:
                Updated:
                Resolved: