Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-7105

SSH holds onto some sessions.

    XMLWordPrintable

Details

    Description

      Summary

      If you open and close a connection really fast it seems to hold onto lots of them in IoServiceListenerSupport.
      In the wild this is sometimes caused by haproxy health checks.

      Steps to Reproduce

      The problem can be reproduced running this python script:

      import socket
      import sys
      
      TCP_IP = '127.0.0.1'
      TCP_PORT = 7999
      
      def makeTestConnection():
              s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
              s.connect((TCP_IP, TCP_PORT))
              s.close()
      
      numConnections = 10000
      if len(sys.argv) == 2:
              numConnections = int(sys.argv[1])
      
      for i in range(numConnections):
              print(str(i))
              makeTestConnection()
      

      Expected Results

      The session object number is small.

      Actual Results

      If you take a heap dump afterwards you will see lots of session objects.
      There seems to be a 2 minute auth timeout that gets hit and some of the sessions are removed but the majority are not removed.
      Creating 10000 connections in seems to result in around 7000 of them hanging around.

      Eclipse Memory Analyzer output:

      Class Name                                      | Objects | Shallow Heap | Retained Heap
      -----------------------------------------------------------------------------------------
      org.apache.sshd.server.session.ServerSession    |   6,841 |    1,860,752 |              
      org.apache.sshd.common.io.nio2.Nio2Session      |   6,841 |      492,552 |              
      org.apache.sshd.common.future.DefaultCloseFuture|  13,685 |      437,920 |              
      org.apache.sshd.common.util.Buffer              |   6,841 |      164,184 |              
      -----------------------------------------------------------------------------------------
      

      This condition could lead to an OutOfMemoryError.

      This problem has been raised against Apache SSHD-417 - Memory leak in IoServiceListenerSupport.managedSessions when clients cut connections quickly, which has now been resolved.

      Attachments

        Issue Links

          Activity

            People

              fschroder@atlassian.com Federico Schroder (Inactive)
              sgoodhew Scott Goodhew (Inactive)
              Votes:
              11 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: