Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-10100

OutOfMemoryError in NioSocketSession's DefaultWriteRequestQueue

      Apache MINA enqueues WriteRequest s in an unbounded queue. Access to the queue is limited only on the basis of the channel window size specified by the client (see SSH_MSG_CHANNEL_OPEN in RFC4254 for details on SSH channel windows). When a client specifies a large window size but is unable to read fast enough from the server, too many requests will be enqueued in memory causing an OutOfMemoryError.

      We have seen this issue particularly with TortoiseGit (version 2.4.0.2+) which has a TortoisePlink which in turn relies on Putty. These clients establish a channel window size of 2GB so operations which produce a large amount of data (e.g.: cloning a large repository) will most likely trigger this error.

      Resolution

      The issue has been resolved by introducing a throttling mechanism on SSH session writes.

      The fix is available in the following versions:

      • 4.14.10, 5.0.8, 5.1.6, 5.2.4, 5.3.2, 5.4.+ and any higher version

      Diagnosis

      • Enable the collection of a Heap Dump when the OutOfMemory is reached by configuring the JVM_SUPPORT_RECOMMENDED_ARGS option (a restart is required for this configuration to take effect):
        JVM_SUPPORT_RECOMMENDED_ARGS="-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/path/to/heap/dump"
        
      • Review the heap dump using a memory analyzer tool (for example Memory Analyzer (MAT)) and check which object is consuming the majority of the memory
        • Bitbucket Server is affected by this if the memory is being taken up by the NioSocketSession and within that object the DefaultWriteRequestQueue
      • Look through the heap dump for instances of the Window class and check its maxSize value to see if it's larger than 2*32*64 (this is the OpenSSH default and usually with TortoiseGit/Putty it will be set at 2GBs).

            [BSERV-10100] OutOfMemoryError in NioSocketSession's DefaultWriteRequestQueue

            Mike Friedrich added a comment - See here: https://git.tartarus.org/?p=simon/putty.git;a=commitdiff;h=16cbd4f26095627796e2774e0ca43648f9c4c692

            Hi awilliams1288985735,
            It can be triggered by any client which uses a large channel window size.

            This issue was discovered after a change in Putty which started setting the window size to 2GB. It seems like Jenkins has some dependency on Putty, but beyond that I don't know if Jenkins is triggering this OutOfMemoryError. Have you followed the steps listed in this ticket to diagnose the issue?

            If you are running a vulnerable instance I would recommend you upgrade your system.

            Regards
            Juan Palacios

            Juan Palacios (Inactive) added a comment - Hi awilliams1288985735 , It can be triggered by any client which uses a large channel window size. This issue was discovered after a change in Putty which started setting the window size to 2GB. It seems like Jenkins has some dependency on Putty , but beyond that I don't know if Jenkins is triggering this OutOfMemoryError. Have you followed the steps listed in this ticket to diagnose the issue? If you are running a vulnerable instance I would recommend you upgrade your system. Regards Juan Palacios

            Can this happen with jenkins SSH requests?

            Alan Williams added a comment - Can this happen with jenkins SSH requests?

            This page contains also suggestions on how to fix it:

            A Sudden SSH out-of-memory http://jonnyzzz.com/blog/2017/08/23/ssh-oom/

            Mike Friedrich added a comment - This page contains also suggestions on how to fix it: A Sudden SSH out-of-memory  http://jonnyzzz.com/blog/2017/08/23/ssh-oom/

              jpalacios Juan Palacios (Inactive)
              jpalacios Juan Palacios (Inactive)
              Affected customers:
              3 This affects my team
              Watchers:
              14 Start watching this issue

                Created:
                Updated:
                Resolved: