OutOfMemoryError in NioSocketSession's DefaultWriteRequestQueue

Apache MINA enqueues WriteRequest s in an unbounded queue. Access to the queue is limited only on the basis of the channel window size specified by the client (see SSH_MSG_CHANNEL_OPEN in RFC4254 for details on SSH channel windows). When a client specifies a large window size but is unable to read fast enough from the server, too many requests will be enqueued in memory causing an OutOfMemoryError.

We have seen this issue particularly with TortoiseGit (version 2.4.0.2+) which has a TortoisePlink which in turn relies on Putty. These clients establish a channel window size of 2GB so operations which produce a large amount of data (e.g.: cloning a large repository) will most likely trigger this error.

Resolution

The issue has been resolved by introducing a throttling mechanism on SSH session writes.

The fix is available in the following versions:

• 4.14.10, 5.0.8, 5.1.6, 5.2.4, 5.3.2, 5.4.+ and any higher version

Diagnosis

• Enable the collection of a Heap Dump when the OutOfMemory is reached by configuring the JVM_SUPPORT_RECOMMENDED_ARGS option (a restart is required for this configuration to take effect):

  JVM_SUPPORT_RECOMMENDED_ARGS="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/heap/dump"
  

• Review the heap dump using a memory analyzer tool (for example Memory Analyzer (MAT)) and check which object is consuming the majority of the memory
  • Bitbucket Server is affected by this if the memory is being taken up by the NioSocketSession and within that object the DefaultWriteRequestQueue
• Look through the heap dump for instances of the Window class and check its maxSize value to see if it's larger than 2*32*64 (this is the OpenSSH default and usually with TortoiseGit/Putty it will be set at 2GBs).