Details
-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
7.2.8, 8.5.6, 8.13.19, 9.4.4
-
7.02
-
36
-
Severity 2 - Major
-
11
-
Description
Summary
The processing of the outgoing mail queue is single threaded and can get delayed (making the notifications to pile up in the queue) due to the thread (processing the notifications) taking a long time to calculate what had changed in an issue before the notification is sent out.
In a few scenarios, the mail job is getting completely stuck and is causing no notifications to be sent out from Jira unless we do a restart.
See also other related issue: JRASERVER-61543
Environment
I have reproduced the issue in:
- JIRA 7.2.8
- Outgoing mail enabled
Steps to Reproduce
- Enable outgoing mail
- Create a test project
- Create some issues to the project inserting in any of the multi-text fields a very long text (approximately 300000 characters).
- Edit one or more of the issues created by deleting the 300000 from the multi-text field and just pasting the first 1000 characters (from the long text of 300000 characters)
- When the notification is being sent the thread processing it will be take some time to calculate the difference of what change in the issue and will have the same stack trace as the one detailed in the description of the bug above.
Expected Results
The thread processing the notifications should process the differences faster or it should have a time out that can be configured by the administrator so when it takes a certain time to process a notification a template notification (such as "Too many things have change in this issue" or similar) is sent instead of spending a long time in calculating the changes.
The administrator should be able to switch the time out on/off as well as being able to set the max time out interval.
Actual Results
The processing of the outgoing mail queue gets delayed, making the notifications to pile up in the queue and affecting the performance of it.
The thread processing the notification will have the below stack trace:
"Sending mailitem com.atlassian.jira.mail.IssueMailQueueItem@682a8028[issue=com.atlassian.jira.issue.IssueImpl@5122ec89[id=837382,summary=Lorem ipsum dolor sit amet, consectetur adipiscing elit ,key=PROJ-123,created=2017-01-06 19:27:14.107,updated=2017-08-18 05:50:47.763,assignee=user1(user1),reporter=user2(user2)],remoteUser=user3(user3),notificationType=Current_Assignee,eventTypeId=2,templateId=2]" #182 daemon prio=5 os_prio=0 tid=0x0000000058c36800 nid=0x1600 runnable [0x000000006631b000] java.lang.Thread.State: RUNNABLE at org.apache.commons.jrcs.diff.myers.MyersDiff.buildPath(MyersDiff.java:129) at org.apache.commons.jrcs.diff.myers.MyersDiff.diff(MyersDiff.java:93) at org.apache.commons.jrcs.diff.Diff.diff(Diff.java:197) at com.atlassian.diff.WordLevelDiffer.diffWords(WordLevelDiffer.java:101) at com.atlassian.diff.WordLevelDiffer.diffLine(WordLevelDiffer.java:91) at com.atlassian.diff.DiffViewBean.createWordLevelDiff(DiffViewBean.java:108) at com.atlassian.jira.mail.DiffUtils.diff(DiffUtils.java:19) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:385) at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:374) at com.atlassian.velocity.htmlsafe.introspection.UnboxingMethod.invoke(UnboxingMethod.java:30) ... at com.atlassian.velocity.DefaultVelocityManager.writeEncodedBodyForContent(DefaultVelocityManager.java:86) at com.atlassian.jira.template.velocity.DefaultVelocityTemplatingEngine$DefaultRenderRequest.toWriterImpl(DefaultVelocityTemplatingEngine.java:129) at com.atlassian.jira.template.velocity.DefaultVelocityTemplatingEngine$DefaultRenderRequest.asPlainText(DefaultVelocityTemplatingEngine.java:108) at com.atlassian.jira.template.velocity.DefaultVelocityTemplatingEngine$DefaultRenderRequest$1.with(DefaultVelocityTemplatingEngine.java:92) at com.atlassian.jira.template.velocity.DefaultVelocityTemplatingEngine$DefaultRenderRequest$StringRepresentation.toString(DefaultVelocityTemplatingEngine.java:77) at com.atlassian.jira.template.velocity.DefaultVelocityTemplatingEngine$DefaultRenderRequest.asPlainText(DefaultVelocityTemplatingEngine.java:94) at com.atlassian.jira.mail.builder.EmailRenderer.renderEmailBody(EmailRenderer.java:103) at com.atlassian.jira.mail.builder.EmailRenderer.render(EmailRenderer.java:150) at com.atlassian.jira.mail.builder.EmailBuilder.renderNow(EmailBuilder.java:155) at com.atlassian.jira.mail.builder.EmailBuilder.renderNowAsQueueItem(EmailBuilder.java:145) at com.atlassian.jira.mail.MailingListCompiler$1.evaluateEmailForRecipient(MailingListCompiler.java:336) at com.atlassian.jira.mail.NotificationRecipientProcessor.evaluateEmails(NotificationRecipientProcessor.java:39) at com.atlassian.jira.mail.MailingListCompiler.evaluateEmails(MailingListCompiler.java:267) ... at com.atlassian.jira.mail.MailingListCompiler.prepareEmail(MailingListCompiler.java:170) at com.atlassian.jira.mail.MailingListCompiler.sendLists(MailingListCompiler.java:101) at com.atlassian.jira.mail.IssueMailQueueItem.send(IssueMailQueueItem.java:128) at com.atlassian.mail.queue.MailQueueImpl.sendBufferUnderLock(MailQueueImpl.java:103) at com.atlassian.mail.queue.MailQueueImpl.sendBuffer(MailQueueImpl.java:56) at com.atlassian.jira.mail.JiraMailQueue$1.apply(JiraMailQueue.java:51) at com.atlassian.jira.mail.JiraMailQueue$1.apply(JiraMailQueue.java:48) at com.atlassian.jira.util.velocity.DefaultVelocityRequestContextFactory.runWithStaticBaseUrl(DefaultVelocityRequestContextFactory.java:110) at com.atlassian.jira.util.DefaultBaseUrl.runWithStaticBaseUrl(DefaultBaseUrl.java:50) at com.atlassian.jira.mail.JiraMailQueue.sendBuffer(JiraMailQueue.java:48) at com.atlassian.jira.service.services.mail.MailQueueService.run(MailQueueService.java:21) ... at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:34) at java.lang.Thread.run(Thread.java:745)
Workaround
Flushing the queue can help to ease the problem as it add another thread to process notifications.
Please note that you shouldn't flush the queue more then once, since it will use more Caesium (scheduler) threads.
Note : The workaround will not help if the Mail process is getting completely stuck, in this case Restart of Jira would be required for the notifications to be sent again.
Additional Notes
It is very important to note that the bigger the difference between the original text in the multi-line text field and what it was updated then the longer the diff libraries will take to process what it was change.
For example comparing 1000 characters vs 300000 characters will take longer than comparing 240000 characters vs 300000 characters as the difference is smaller in the second comparison.
