Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-77271

After upgrading to 9.10.1 from 9.7.1 separate connections for each email notifications to Mail Server are concurrently established.

XMLWordPrintable

      Issue Summary

      After upgrading to 9.10.1 from 9.7.1 separate connections for each email notification to the Mail Server are concurrently established.

      This is reproducible in Data Center
      Reproducible at Customer: Yes/No - Yes
      Reproducible at Support: Yes/No - No

      The issue has been reported after upgrading to 9.10.1 an error has been observed and a massive number of errors were from the SMTP with the following content:

      Caused by: javax.mail.MessagingException: Got bad greeting from SMTP host: hostname@domain.com, port: 25, response: 421 4.3.2 The maximum number of concurrent server connections has exceeded a per-source limit, closing transmission channel (<hostname>.<domain>)
      

      A Socket Statistics dump was obtained from the Linux host where JIRA was running and during the mail-sending process it can be observed multiple connections to the SMTP server are made. Attached is a sample SS output (20230825-jira-ss-dump.txt) showing the concurrent connections.

      From the atlassian-jira.log file

      2023-08-25 13:56:02,155+0100 DEBUG [] Sending mailitem To='USERABC@domain.com' Subject='TEST SUBJECT A' From='null' FromName='TESTUSERFROM' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='<JIRA.711085.1690996633000@Atlassian.JIRA>' MimeType='text/html' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@18112a1b' MessageId='null' ExcludeSubjectPrefix=false' anonymous    Mail Queue Service [c.atlassian.mail.outgoing] Message was sent with Message-Id <JIRA.711085.1690996633000.160850.1692968162002@Atlassian.JIRA>
      
      2023-08-25 13:56:02,298+0100 DEBUG [] Sending mailitem To='USERDEF@domain.com' Subject='TEST SUBJECT A' From='null' FromName='TESTUSERFROM' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='<JIRA.711085.1690996633000@Atlassian.JIRA>' MimeType='text/html' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@2743a394' MessageId='null' ExcludeSubjectPrefix=false' anonymous    Mail Queue Service [c.atlassian.mail.outgoing] Message was sent with Message-Id <JIRA.711085.1690996633000.160851.1692968162155@Atlassian.JIRA>
      

      From the outgoing-jira.log file we could identify that in a certain time window, we can see that there are 20 messages being sent out one after the other and then a failure is observed.

      2023-08-30 23:19:02,002+0100 Getting transport for protocol [smtp]
      2023-08-30 23:19:02,003+0100 Obtaining transport object directly (no caching).
      2023-08-30 23:19:02,003+0100 Got transport: [smtp://oracleas@mailrelay.domain]. Connecting
      2023-08-30 23:19:02,014+0100 Sending message
      
      # 2023-08-30 23:19:02,156+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1346.1693433942002@Atlassian.JIRA>
      # 2023-08-30 23:19:02,322+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1347.1693433942156@Atlassian.JIRA>
      # 2023-08-30 23:19:02,478+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1348.1693433942322@Atlassian.JIRA>
      # 2023-08-30 23:19:02,632+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1349.1693433942478@Atlassian.JIRA>
      # 2023-08-30 23:19:02,794+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1350.1693433942632@Atlassian.JIRA>
      # 2023-08-30 23:19:02,936+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1351.1693433942794@Atlassian.JIRA>
      # 2023-08-30 23:19:03,075+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1352.1693433942936@Atlassian.JIRA>
      # 2023-08-30 23:19:03,225+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1353.1693433943075@Atlassian.JIRA>
      # 2023-08-30 23:19:03,367+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1354.1693433943225@Atlassian.JIRA>
      # 2023-08-30 23:19:03,525+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1355.1693433943367@Atlassian.JIRA>
      # 2023-08-30 23:19:03,668+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1356.1693433943526@Atlassian.JIRA>
      # 2023-08-30 23:19:03,806+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1357.1693433943669@Atlassian.JIRA>
      # 2023-08-30 23:19:03,955+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1358.1693433943807@Atlassian.JIRA>
      # 2023-08-30 23:19:04,101+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1359.1693433943955@Atlassian.JIRA>
      # 2023-08-30 23:19:04,245+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1360.1693433944101@Atlassian.JIRA>
      # 2023-08-30 23:19:04,385+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1361.1693433944245@Atlassian.JIRA>
      # 2023-08-30 23:19:04,529+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1362.1693433944385@Atlassian.JIRA>
      # 2023-08-30 23:19:04,680+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1363.1693433944529@Atlassian.JIRA>
      # 2023-08-30 23:19:04,822+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1364.1693433944680@Atlassian.JIRA>
      # 2023-08-30 23:19:04,964+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1365.1693433944822@Atlassian.JIRA>
      

      And then it is followed by the error as below

      2023-08-30 23:19:04,966+0100 ERROR [] Sending mailitem To='<username>@domain' Subject='ISSUE_ID AND SUBJECT' From='null' FromName='Username' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='<JIRA.640934.1671198267000@Atlassian.JIRA>' MimeType='text/html' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@5edd13d7' MessageId='null' ExcludeSubjectPrefix=false' anonymous    Mail Queue Service [c.a.mail.queue.MailQueueImpl] Error occurred in sending e-mail: To='usrename@domain.com' Subject='ISSUE_ID AND SUBJECT ' From='null' FromName='Username' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='<JIRA.640934.1671198267000@Atlassian.JIRA>' MimeType='text/html' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@5edd13d7' MessageId='null' ExcludeSubjectPrefix=false'
      com.atlassian.mail.MailException: javax.mail.MessagingException: Got bad greeting from SMTP host: hostname@domain.com, port: 25, response: 421 4.3.2 The maximum number of concurrent server connections has exceeded a per-source limit, closing transmission channel (channel_id.domain.com)
      
      Caused by: javax.mail.MessagingException: Got bad greeting from SMTP host: hostname@domain.com, port: 25, response: 421 4.3.2 The maximum number of concurrent server connections has exceeded a per-source limit, closing transmission channel (channel_id.domain.com)
      

      Here are some changes in the atlassian-mail-x.x.x.jar files in both versions
      For Jira v 9.7.1 the file atlassian-mail-5.0.4.jar
      For Jira v 9.10.1 the file atlassian-mail-6.0.1.jar
      Image attached

      Workaround

      (A)  - Increase concurrency permitted by the mail server

      On the mail server side, adjust the MessageRateLimit. For example, to 20 or 100, depending on your mail load.

      See Microsoft article:  Exchange Server Support forum; 421 4.4.2 Message submission rate for this client has exceeded the configured limit.

      (B) - Reduce the concurrency of Jira sending mail to the mail server

      By sending mail more frequently, when Jira does need to send mail, there will likely be less mail and therefore less concurrent mail operations:

      • Reduce the Mail Queue Service on each node (Jira Admin -> System -> Services -> (Mail Queue Service)
        • The schedule is a Java cron expression which can be interpreted with an online helper tool, eg https://www.javainuse.com/cron
        • For example, if your Mail Queue Service is set to every 5 minutes:
          0 0/5 * 1/1 * ? *
          concurrency could be reduced by setting it to every 1 minute:
          0 0/1 * 1/1 * ? *
      • Reduce the batching notification interval
      • Disabling batching notifications
        • This will result in each individual node sending mail according to the Mail Queue Service schedule

              Unassigned Unassigned
              282265999e47 Hameed Saudagar
              Votes:
              25 Vote for this issue
              Watchers:
              58 Start watching this issue

                Created:
                Updated:
                Resolved: