BSERV-10100 introduced throttling of SSH writes when the remote SSH client specifies a large SSH window size. Recent versions of PuTTY request a windows size of 2GB and the throttling was necessary to prevent OutOfMemoryErrors.
Unfortunately, the fix for
BSERV-10100 introduced a race condition which can result in a deadlock when an SSH connection is terminated by the remote client while the pending writes to the client are throttled. This deadlock affects the SSH I/O processing thread(s). SSH sessions are mapped to I/O processing threads. Any (new) sessions mapped to a deadlocked I/O processing thread will hang without any I/O processing.
Some git operations over SSH will hang without any output and fail after a timeout occurs.
The atlassian-bitbucket.log contains "Too many open files" errors:
Take a thread dump of the Bitbucket Server instance and look for threads with a name starting with NioProcessor-. If these threads have stacktrace similar to the following, the thread is deadlocked.