• Type: Bug
• Resolution: Fixed
• Priority: High
• Fix Version/s: 5.3.7, 5.5.6, 5.6.3, 5.7.1, 4.14.12, 5.0.10, 5.1.9, 5.2.8, 5.4.6
• Affects Version/s: 5.4.0, 4.14.10, 5.0.8, 5.1.6, 5.2.4, 5.3.2, 5.7.0
• Component/s: SSH
• Labels:

  None

[BSERV-10575] Too many open files caused by a deadlock in SSH threads due to race condition with PuTTY SSH clients

Owen made changes - 10/Apr/2019 6:47 AM

Workflow

Original: Stash Workflow - Restricted [ 2517053 ]

New: JAC Bug Workflow v3 [ 3137264 ]

Owen made changes - 24/Oct/2018 5:06 AM

Symptom Severity

Original: Major [ 14431 ]

New: Severity 2 - Major [ 15831 ]

Frank Doherty made changes - 17/Apr/2018 11:32 PM

Remote Link

New: This issue links to "Page (Extranet)" [ 363088 ]

Frank Doherty made changes - 17/Apr/2018 11:32 PM

Remote Link

New: This issue links to "SSP-28756 (Atlassian Support System)" [ 363119 ]

Andre Marques (Inactive) made changes - 15/Feb/2018 7:45 PM

Remote Link

New: This issue links to "PS-22298 (Atlassian Support System)" [ 351406 ]

Caterina Curti made changes - 13/Feb/2018 2:32 AM

Description

Original: BSERV-10100 introduced throttling of SSH writes when the remote SSH client specifies a large SSH window size. Recent versions of PuTTY request a windows size of 2GB and the throttling was necessary to prevent {{OutOfMemoryErrors}}. Unfortunately, the fix for BSERV-10100 introduced a race condition which can result in a deadlock when an SSH connection is terminated by the remote client while the pending writes to the client are throttled. This deadlock affects the SSH I/O processing thread(s). SSH sessions are mapped to I/O processing threads. Any (new) sessions mapped to a deadlocked I/O processing thread will hang without any I/O processing. *Symptoms* Some git operations over SSH will hang without any output and fail after a timeout occurs. The {{atlassian-bitbucket.log}} contains "Too many open files" errors: {noformat} ERROR [https-jsse-nio-8443-Acceptor-0] o.apache.tomcat.util.net.NioEndpoint Socket accept failed java.io.IOException: Too many open files {noformat} *Diagnosis* Take a thread dump of the Bitbucket Server instance and look for threads with a name starting with {{NioProcessor-}}. If these threads have stacktrace similar to the following, the thread is deadlocked. {noformat}  - org.apache.sshd.common.channel.ChannelOutputStream.close() @bci=0, line=232 (Compiled frame)  - org.apache.sshd.common.util.io.IoUtils.closeQuietly(java.io.Closeable[]) @bci=29, line=127 (Compiled frame)  - org.apache.sshd.server.channel.ChannelSession.doCloseImmediately() @bci=155, line=209 (Compiled frame)  - org.apache.sshd.common.util.closeable.AbstractCloseable.close(boolean) @bci=65, line=81 (Compiled frame)  - org.apache.sshd.common.channel.AbstractChannel.close(boolean) @bci=37, line=448 (Compiled frame)  - com.atlassian.bitbucket.internal.ssh.server.ScmHostingChannelSession.close(boolean) @bci=2, line=47 (Compiled frame)  - org.apache.sshd.common.util.closeable.ParallelCloseable.doClose(boolean) @bci=117, line=65 (Compiled frame)  - org.apache.sshd.common.util.closeable.SimpleCloseable.close(boolean) @bci=14, line=63 (Compiled frame)  - org.apache.sshd.common.util.closeable.AbstractInnerCloseable.doCloseImmediately() @bci=5, line=47 (Compiled frame) {noformat}

New: BSERV-10100 introduced throttling of SSH writes when the remote SSH client specifies a large SSH window size. Recent versions of PuTTY request a windows size of 2GB and the throttling was necessary to prevent {{OutOfMemoryErrors}}. Unfortunately, the fix for BSERV-10100 introduced a race condition which can result in a deadlock when an SSH connection is terminated by the remote client while the pending writes to the client are throttled. This deadlock affects the SSH I/O processing thread(s). SSH sessions are mapped to I/O processing threads. Any (new) sessions mapped to a deadlocked I/O processing thread will hang without any I/O processing. *Symptoms* Some git operations over SSH will hang without any output and fail after a timeout occurs. The {{atlassian-bitbucket.log}} contains "Too many open files" errors: {noformat} java.io.IOException: Too many open files {noformat} *Diagnosis* Take a thread dump of the Bitbucket Server instance and look for threads with a name starting with {{NioProcessor-}}. If these threads have stacktrace similar to the following, the thread is deadlocked. {noformat}  - org.apache.sshd.common.channel.ChannelOutputStream.close() @bci=0, line=232 (Compiled frame)  - org.apache.sshd.common.util.io.IoUtils.closeQuietly(java.io.Closeable[]) @bci=29, line=127 (Compiled frame)  - org.apache.sshd.server.channel.ChannelSession.doCloseImmediately() @bci=155, line=209 (Compiled frame)  - org.apache.sshd.common.util.closeable.AbstractCloseable.close(boolean) @bci=65, line=81 (Compiled frame)  - org.apache.sshd.common.channel.AbstractChannel.close(boolean) @bci=37, line=448 (Compiled frame)  - com.atlassian.bitbucket.internal.ssh.server.ScmHostingChannelSession.close(boolean) @bci=2, line=47 (Compiled frame)  - org.apache.sshd.common.util.closeable.ParallelCloseable.doClose(boolean) @bci=117, line=65 (Compiled frame)  - org.apache.sshd.common.util.closeable.SimpleCloseable.close(boolean) @bci=14, line=63 (Compiled frame)  - org.apache.sshd.common.util.closeable.AbstractInnerCloseable.doCloseImmediately() @bci=5, line=47 (Compiled frame) {noformat}

Caterina Curti made changes - 13/Feb/2018 2:25 AM

Description

Original: BSERV-10100 introduced throttling of SSH writes when the remote SSH client specifies a large SSH window size. Recent versions of PuTTY request a windows size of 2GB and the throttling was necessary to prevent {{OutOfMemoryErrors}}. Unfortunately, the fix for BSERV-10100 introduced a race condition which can result in a deadlock when an SSH connection is terminated by the remote client while the pending writes to the client are throttled. This deadlock affects the SSH I/O processing thread(s). SSH sessions are mapped to I/O processing threads. Any (new) sessions mapped to a deadlocked I/O processing thread will hang without any I/O processing. *Symptoms* Some git operations over SSH will hang without any output and fail after a timeout occurs. *Diagnosis* Take a thread dump of the Bitbucket Server instance and look for threads with a name starting with {{NioProcessor-}}. If these threads have stacktrace similar to the following, the thread is deadlocked. {noformat}  - org.apache.sshd.common.channel.ChannelOutputStream.close() @bci=0, line=232 (Compiled frame)  - org.apache.sshd.common.util.io.IoUtils.closeQuietly(java.io.Closeable[]) @bci=29, line=127 (Compiled frame)  - org.apache.sshd.server.channel.ChannelSession.doCloseImmediately() @bci=155, line=209 (Compiled frame)  - org.apache.sshd.common.util.closeable.AbstractCloseable.close(boolean) @bci=65, line=81 (Compiled frame)  - org.apache.sshd.common.channel.AbstractChannel.close(boolean) @bci=37, line=448 (Compiled frame)  - com.atlassian.bitbucket.internal.ssh.server.ScmHostingChannelSession.close(boolean) @bci=2, line=47 (Compiled frame)  - org.apache.sshd.common.util.closeable.ParallelCloseable.doClose(boolean) @bci=117, line=65 (Compiled frame)  - org.apache.sshd.common.util.closeable.SimpleCloseable.close(boolean) @bci=14, line=63 (Compiled frame)  - org.apache.sshd.common.util.closeable.AbstractInnerCloseable.doCloseImmediately() @bci=5, line=47 (Compiled frame) {noformat}

New: BSERV-10100 introduced throttling of SSH writes when the remote SSH client specifies a large SSH window size. Recent versions of PuTTY request a windows size of 2GB and the throttling was necessary to prevent {{OutOfMemoryErrors}}. Unfortunately, the fix for BSERV-10100 introduced a race condition which can result in a deadlock when an SSH connection is terminated by the remote client while the pending writes to the client are throttled. This deadlock affects the SSH I/O processing thread(s). SSH sessions are mapped to I/O processing threads. Any (new) sessions mapped to a deadlocked I/O processing thread will hang without any I/O processing. *Symptoms* Some git operations over SSH will hang without any output and fail after a timeout occurs. The {{atlassian-bitbucket.log}} contains "Too many open files" errors: {noformat} ERROR [https-jsse-nio-8443-Acceptor-0] o.apache.tomcat.util.net.NioEndpoint Socket accept failed java.io.IOException: Too many open files {noformat} *Diagnosis* Take a thread dump of the Bitbucket Server instance and look for threads with a name starting with {{NioProcessor-}}. If these threads have stacktrace similar to the following, the thread is deadlocked. {noformat}  - org.apache.sshd.common.channel.ChannelOutputStream.close() @bci=0, line=232 (Compiled frame)  - org.apache.sshd.common.util.io.IoUtils.closeQuietly(java.io.Closeable[]) @bci=29, line=127 (Compiled frame)  - org.apache.sshd.server.channel.ChannelSession.doCloseImmediately() @bci=155, line=209 (Compiled frame)  - org.apache.sshd.common.util.closeable.AbstractCloseable.close(boolean) @bci=65, line=81 (Compiled frame)  - org.apache.sshd.common.channel.AbstractChannel.close(boolean) @bci=37, line=448 (Compiled frame)  - com.atlassian.bitbucket.internal.ssh.server.ScmHostingChannelSession.close(boolean) @bci=2, line=47 (Compiled frame)  - org.apache.sshd.common.util.closeable.ParallelCloseable.doClose(boolean) @bci=117, line=65 (Compiled frame)  - org.apache.sshd.common.util.closeable.SimpleCloseable.close(boolean) @bci=14, line=63 (Compiled frame)  - org.apache.sshd.common.util.closeable.AbstractInnerCloseable.doCloseImmediately() @bci=5, line=47 (Compiled frame) {noformat}

Caterina Curti made changes - 13/Feb/2018 2:23 AM

Summary

Original: Deadlock in SSH threads due to race condition with PuTTY SSH clients

New: Too many open files caused by a deadlock in SSH threads due to race condition with PuTTY SSH clients

Rafael Donato made changes - 06/Feb/2018 11:02 PM

Remote Link

New: This issue links to "Page (Extranet)" [ 349710 ]

Christian Glockner made changes - 05/Feb/2018 10:15 AM

Remote Link

New: This issue links to "Page (Extranet)" [ 349382 ]

Load more older events