-
Bug
-
Resolution: Unresolved
-
Medium
-
None
-
7.2.8, 8.5.6, 8.13.19, 9.4.4
-
7.02
-
39
-
Severity 2 - Major
-
10
-
Summary
The processing of the outgoing mail queue is single threaded and can get delayed (making the notifications to pile up in the queue) due to the thread (processing the notifications) taking a long time to calculate what had changed in an issue before the notification is sent out.
In a few scenarios, the mail job is getting completely stuck and is causing no notifications to be sent out from Jira unless we do a restart.
See also other related issue: JRASERVER-61543
Environment
I have reproduced the issue in:
- JIRA 7.2.8
- Outgoing mail enabled
Steps to Reproduce
- Enable outgoing mail
- Create a test project
- Create some issues to the project inserting in any of the multi-text fields a very long text (approximately 300000 characters).
- Edit one or more of the issues created by deleting the 300000 from the multi-text field and just pasting the first 1000 characters (from the long text of 300000 characters)
- When the notification is being sent the thread processing it will be take some time to calculate the difference of what change in the issue and will have the same stack trace as the one detailed in the description of the bug above.
Expected Results
The thread processing the notifications should process the differences faster or it should have a time out that can be configured by the administrator so when it takes a certain time to process a notification a template notification (such as "Too many things have change in this issue" or similar) is sent instead of spending a long time in calculating the changes.
The administrator should be able to switch the time out on/off as well as being able to set the max time out interval.
Actual Results
The processing of the outgoing mail queue gets delayed, making the notifications to pile up in the queue and affecting the performance of it.
The thread processing the notification will have the below stack trace:
"Sending mailitem com.atlassian.jira.mail.IssueMailQueueItem@682a8028[issue=com.atlassian.jira.issue.IssueImpl@5122ec89[id=837382,summary=Lorem ipsum dolor sit amet, consectetur adipiscing elit ,key=PROJ-123,created=2017-01-06 19:27:14.107,updated=2017-08-18 05:50:47.763,assignee=user1(user1),reporter=user2(user2)],remoteUser=user3(user3),notificationType=Current_Assignee,eventTypeId=2,templateId=2]" #182 daemon prio=5 os_prio=0 tid=0x0000000058c36800 nid=0x1600 runnable [0x000000006631b000] java.lang.Thread.State: RUNNABLE at org.apache.commons.jrcs.diff.myers.MyersDiff.buildPath(MyersDiff.java:129) at org.apache.commons.jrcs.diff.myers.MyersDiff.diff(MyersDiff.java:93) at org.apache.commons.jrcs.diff.Diff.diff(Diff.java:197) at com.atlassian.diff.WordLevelDiffer.diffWords(WordLevelDiffer.java:101) at com.atlassian.diff.WordLevelDiffer.diffLine(WordLevelDiffer.java:91) at com.atlassian.diff.DiffViewBean.createWordLevelDiff(DiffViewBean.java:108) at com.atlassian.jira.mail.DiffUtils.diff(DiffUtils.java:19) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:385) at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:374) at com.atlassian.velocity.htmlsafe.introspection.UnboxingMethod.invoke(UnboxingMethod.java:30) ... at com.atlassian.velocity.DefaultVelocityManager.writeEncodedBodyForContent(DefaultVelocityManager.java:86) at com.atlassian.jira.template.velocity.DefaultVelocityTemplatingEngine$DefaultRenderRequest.toWriterImpl(DefaultVelocityTemplatingEngine.java:129) at com.atlassian.jira.template.velocity.DefaultVelocityTemplatingEngine$DefaultRenderRequest.asPlainText(DefaultVelocityTemplatingEngine.java:108) at com.atlassian.jira.template.velocity.DefaultVelocityTemplatingEngine$DefaultRenderRequest$1.with(DefaultVelocityTemplatingEngine.java:92) at com.atlassian.jira.template.velocity.DefaultVelocityTemplatingEngine$DefaultRenderRequest$StringRepresentation.toString(DefaultVelocityTemplatingEngine.java:77) at com.atlassian.jira.template.velocity.DefaultVelocityTemplatingEngine$DefaultRenderRequest.asPlainText(DefaultVelocityTemplatingEngine.java:94) at com.atlassian.jira.mail.builder.EmailRenderer.renderEmailBody(EmailRenderer.java:103) at com.atlassian.jira.mail.builder.EmailRenderer.render(EmailRenderer.java:150) at com.atlassian.jira.mail.builder.EmailBuilder.renderNow(EmailBuilder.java:155) at com.atlassian.jira.mail.builder.EmailBuilder.renderNowAsQueueItem(EmailBuilder.java:145) at com.atlassian.jira.mail.MailingListCompiler$1.evaluateEmailForRecipient(MailingListCompiler.java:336) at com.atlassian.jira.mail.NotificationRecipientProcessor.evaluateEmails(NotificationRecipientProcessor.java:39) at com.atlassian.jira.mail.MailingListCompiler.evaluateEmails(MailingListCompiler.java:267) ... at com.atlassian.jira.mail.MailingListCompiler.prepareEmail(MailingListCompiler.java:170) at com.atlassian.jira.mail.MailingListCompiler.sendLists(MailingListCompiler.java:101) at com.atlassian.jira.mail.IssueMailQueueItem.send(IssueMailQueueItem.java:128) at com.atlassian.mail.queue.MailQueueImpl.sendBufferUnderLock(MailQueueImpl.java:103) at com.atlassian.mail.queue.MailQueueImpl.sendBuffer(MailQueueImpl.java:56) at com.atlassian.jira.mail.JiraMailQueue$1.apply(JiraMailQueue.java:51) at com.atlassian.jira.mail.JiraMailQueue$1.apply(JiraMailQueue.java:48) at com.atlassian.jira.util.velocity.DefaultVelocityRequestContextFactory.runWithStaticBaseUrl(DefaultVelocityRequestContextFactory.java:110) at com.atlassian.jira.util.DefaultBaseUrl.runWithStaticBaseUrl(DefaultBaseUrl.java:50) at com.atlassian.jira.mail.JiraMailQueue.sendBuffer(JiraMailQueue.java:48) at com.atlassian.jira.service.services.mail.MailQueueService.run(MailQueueService.java:21) ... at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:34) at java.lang.Thread.run(Thread.java:745)
Workaround
Manual Intervention
Flushing the mail queue can help to ease the problem, as the flush will occur on a different thread to the slow thread performing the diff, and thus create a second opportunity for mail sending. If that second thread encounters another large string to diff, then that mail flush thread is expected to get stuck - not ideal.
Remove string comparison (diff) from email content
Email content in Jira is controlled by template files. These templates are located in Jira's home directory:
- Batching ON: JIRA_HOME/data/templates/email-batch
- Batching OFF: JIRA_HOME/data/templates/email
There are several files where the problematic string comparison (diff) occurs:
- ./email-batch/html/IssueUpdateBatcher-content.vm
- ./email/html/includes/changelog-issue-description.vm
- ./email/html/includes/fields/changelog.vm
- ./email/html/issuecommentedited.vm
The following instructions remove .diff for comment and field changes and replace with the text "Content updated"
./email/html/includes/changelog-issue-description.vm Line 12:
#set($textParagraph = $!diffutils.diff($oldText, "background-color: ${auiErrorBackgroundColour}; text-decoration:line-through;", $newText, "background-color:${auiSuccessBackgroundColour};"))
Replace with:
#set($textParagraph = "Content updated")
./email/html/includes/fields/changelog.vm Line 36:
$!diffutils.diff($oldText, "background-color:${auiErrorBackgroundColour};text-decoration:line-through;", $newText, "background-color:${auiSuccessBackgroundColour};")
Replace with:
Content updated
./email/html/issuecommentedited.vm Line 11:
#set ($htmlComment = $!diffutils.diff($originalcomment.body, "background-color:${auiErrorBackgroundColour};text-decoration:line-through;", $comment.body, "background-color:${auiSuccessBackgroundColour};"))
Replace with:
#set ($htmlComment = "Content updated")
./email-batch/html/IssueUpdateBatcher-content.vm Line 83:
#if($update.iconUrl)<img src="$update.iconUrl" height="16" width="16" border="0" /> #end$update.diff
Replace with:
#if($update.iconUrl)<img src="$update.iconUrl" height="16" width="16" border="0" /> #end Content updated
./email-batch/html/IssueUpdateBatcher-content.vm Line 138:
<td valign="top" align="left" class="updates-diff-content">$update.diff</td>
Replace with:
<td valign="top" align="left" class="updates-diff-content">Content updated</td>
./email-batch/html/IssueUpdateBatcher-content.vm Line 262:
$comment.diff
Replace with:
Content updated
./email-batch/html/IssueUpdateBatcher-content.vm Line 321:
<td valign="top" align="left" class="updates-diff-content">$update.diff</td>
Replace with:
<td valign="top" align="left" class="updates-diff-content">Content updated</td>
End Result: Instead of the differential comparison showing, instead, the text "Content updated" will show. As the diff is not performed, email sending should proceed as normal.
Additional Notes
It is very important to note that the bigger the difference between the original text in the multi-line text field and what it was updated then the longer the diff libraries will take to process what it was change.
For example comparing 1000 characters vs 300000 characters will take longer than comparing 240000 characters vs 300000 characters as the difference is smaller in the second comparison.
