Details
-
Bug
-
Resolution: Done
-
Highest
-
9.11.0, 9.10.1, 9.12.0, 9.9.2, 9.12.2
-
9.09
-
11
-
Severity 2 - Major
-
46
-
Description
Issue Summary
After upgrading to 9.10.1 from 9.7.1 separate connections for each email notification to the Mail Server are concurrently established.
This is reproducible in Data Center
Reproducible at Customer: Yes/No - Yes
Reproducible at Support: Yes/No - No
The issue has been reported after upgrading to 9.10.1 an error has been observed and a massive number of errors were from the SMTP with the following content:
Caused by: javax.mail.MessagingException: Got bad greeting from SMTP host: hostname@domain.com, port: 25, response: 421 4.3.2 The maximum number of concurrent server connections has exceeded a per-source limit, closing transmission channel (<hostname>.<domain>)
A Socket Statistics dump was obtained from the Linux host where JIRA was running and during the mail-sending process it can be observed multiple connections to the SMTP server are made. Attached is a sample SS output (20230825-jira-ss-dump.txt) showing the concurrent connections.
From the atlassian-jira.log file
2023-08-25 13:56:02,155+0100 DEBUG [] Sending mailitem To='USERABC@domain.com' Subject='TEST SUBJECT A' From='null' FromName='TESTUSERFROM' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='<JIRA.711085.1690996633000@Atlassian.JIRA>' MimeType='text/html' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@18112a1b' MessageId='null' ExcludeSubjectPrefix=false' anonymous Mail Queue Service [c.atlassian.mail.outgoing] Message was sent with Message-Id <JIRA.711085.1690996633000.160850.1692968162002@Atlassian.JIRA> 2023-08-25 13:56:02,298+0100 DEBUG [] Sending mailitem To='USERDEF@domain.com' Subject='TEST SUBJECT A' From='null' FromName='TESTUSERFROM' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='<JIRA.711085.1690996633000@Atlassian.JIRA>' MimeType='text/html' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@2743a394' MessageId='null' ExcludeSubjectPrefix=false' anonymous Mail Queue Service [c.atlassian.mail.outgoing] Message was sent with Message-Id <JIRA.711085.1690996633000.160851.1692968162155@Atlassian.JIRA>
From the outgoing-jira.log file we could identify that in a certain time window, we can see that there are 20 messages being sent out one after the other and then a failure is observed.
2023-08-30 23:19:02,002+0100 Getting transport for protocol [smtp] 2023-08-30 23:19:02,003+0100 Obtaining transport object directly (no caching). 2023-08-30 23:19:02,003+0100 Got transport: [smtp://oracleas@mailrelay.domain]. Connecting 2023-08-30 23:19:02,014+0100 Sending message # 2023-08-30 23:19:02,156+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1346.1693433942002@Atlassian.JIRA> # 2023-08-30 23:19:02,322+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1347.1693433942156@Atlassian.JIRA> # 2023-08-30 23:19:02,478+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1348.1693433942322@Atlassian.JIRA> # 2023-08-30 23:19:02,632+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1349.1693433942478@Atlassian.JIRA> # 2023-08-30 23:19:02,794+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1350.1693433942632@Atlassian.JIRA> # 2023-08-30 23:19:02,936+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1351.1693433942794@Atlassian.JIRA> # 2023-08-30 23:19:03,075+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1352.1693433942936@Atlassian.JIRA> # 2023-08-30 23:19:03,225+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1353.1693433943075@Atlassian.JIRA> # 2023-08-30 23:19:03,367+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1354.1693433943225@Atlassian.JIRA> # 2023-08-30 23:19:03,525+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1355.1693433943367@Atlassian.JIRA> # 2023-08-30 23:19:03,668+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1356.1693433943526@Atlassian.JIRA> # 2023-08-30 23:19:03,806+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1357.1693433943669@Atlassian.JIRA> # 2023-08-30 23:19:03,955+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1358.1693433943807@Atlassian.JIRA> # 2023-08-30 23:19:04,101+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1359.1693433943955@Atlassian.JIRA> # 2023-08-30 23:19:04,245+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1360.1693433944101@Atlassian.JIRA> # 2023-08-30 23:19:04,385+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1361.1693433944245@Atlassian.JIRA> # 2023-08-30 23:19:04,529+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1362.1693433944385@Atlassian.JIRA> # 2023-08-30 23:19:04,680+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1363.1693433944529@Atlassian.JIRA> # 2023-08-30 23:19:04,822+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1364.1693433944680@Atlassian.JIRA> # 2023-08-30 23:19:04,964+0100 --> Message was sent with Message-Id <JIRA.640934.1671198267000.1365.1693433944822@Atlassian.JIRA>
And then it is followed by the error as below
2023-08-30 23:19:04,966+0100 ERROR [] Sending mailitem To='<username>@domain' Subject='ISSUE_ID AND SUBJECT' From='null' FromName='Username' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='<JIRA.640934.1671198267000@Atlassian.JIRA>' MimeType='text/html' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@5edd13d7' MessageId='null' ExcludeSubjectPrefix=false' anonymous Mail Queue Service [c.a.mail.queue.MailQueueImpl] Error occurred in sending e-mail: To='usrename@domain.com' Subject='ISSUE_ID AND SUBJECT ' From='null' FromName='Username' Cc='null' Bcc='null' ReplyTo='null' InReplyTo='<JIRA.640934.1671198267000@Atlassian.JIRA>' MimeType='text/html' Encoding='UTF-8' Multipart='javax.mail.internet.MimeMultipart@5edd13d7' MessageId='null' ExcludeSubjectPrefix=false' com.atlassian.mail.MailException: javax.mail.MessagingException: Got bad greeting from SMTP host: hostname@domain.com, port: 25, response: 421 4.3.2 The maximum number of concurrent server connections has exceeded a per-source limit, closing transmission channel (channel_id.domain.com) Caused by: javax.mail.MessagingException: Got bad greeting from SMTP host: hostname@domain.com, port: 25, response: 421 4.3.2 The maximum number of concurrent server connections has exceeded a per-source limit, closing transmission channel (channel_id.domain.com)
Here are some changes in the atlassian-mail-x.x.x.jar files in both versions
For Jira v 9.7.1 the file atlassian-mail-5.0.4.jar
For Jira v 9.10.1 the file atlassian-mail-6.0.1.jar
Image attached
Workaround
(A) - Increase concurrency permitted by the mail server
On the mail server side, adjust the MessageRateLimit. For example, to 20 or 100, depending on your mail load.
See Microsoft article: Exchange Server Support forum; 421 4.4.2 Message submission rate for this client has exceeded the configured limit.
(B) - Reduce the concurrency of Jira sending mail to the mail server
By sending mail more frequently, when Jira does need to send mail, there will likely be less mail and therefore less concurrent mail operations:
- Reduce the Mail Queue Service on each node (Jira Admin -> System -> Services -> (Mail Queue Service)
- The schedule is a Java cron expression which can be interpreted with an online helper tool, eg https://www.javainuse.com/cron
- For example, if your Mail Queue Service is set to every 5 minutes:
0 0/5 * 1/1 * ? *
concurrency could be reduced by setting it to every 1 minute:
0 0/1 * 1/1 * ? *
- Reduce the batching notification interval
- Disabling batching notifications
- This will result in each individual node sending mail according to the Mail Queue Service schedule
