Bitbucket Server may leak SSH sessions under some conditions

Summary

Under some conditions, the Apache Mina SSH server bundled with Bitbucket Server may leak SSH session objects. Under some extreme conditions, this may cause Out Of Memory Errors.

If you open and close a connection really fast it seems to hold onto lots of them in IoServiceListenerSupport.
In the wild this can be caused by haproxy health checks.

Steps to Reproduce

The problem can be reproduced in environments where you're Setting up SSH port forwarding but you left a check port statement on your HAProxy. I.e:

backend your-backend--ssh
        mode tcp
        server your-server-hostname-stash <YOUR_BTIBUCKET_IP>:7999 check port 7999

Expected Results

The session object number in the JVM Heap is small.

Actual Results

If you set up a Heap Dump on an instance that runs out of memory, you would be able to see huge number of objects:

The data below was extracted from an actual customer's heap dump:

Class                                            Objects         Shallow Size       Retained Size
org.apache.sshd.server.session.ServerSession      328234           97157264          359743528
org.apache.sshd.common.future.DefaultCloseFuture  984456           31502592          41998144
org.apache.sshd.common.util.Buffer                328234           7877616           97156992

This condition could lead to an OutOfMemoryError 24-72 hours after your instance has started up.

This problem has been raised against Apache. See DIRMINA-1021 for further information.

Workaround

Remove the statement check port 7999 from your HAProxy configuration.