Uploaded image for project: 'Jira Server and Data Center'
  1. Jira Server and Data Center
  2. JRASERVER-65963

Outgoing mail queue processing is delayed due to thread taking a long time to process what was changed in an issue

    XMLWordPrintable

Details

    Description

      Summary

      The processing of the outgoing mail queue is single threaded and can get delayed (making the notifications to pile up in the queue) due to the thread (processing the notifications) taking a long time to calculate what had changed in an issue before the notification is sent out.
      See also other related issue: JRASERVER-61543

      Environment

      I have reproduced the issue in:

      • JIRA 7.2.8
      • Outgoing mail enabled

       

      Steps to Reproduce

      1. Enable outgoing mail
      2. Create a test project
      3. Create some issues to the project inserting in any of the multi-text fields a very long text (approximately 300000 characters).
      4. Edit one or more of the issues created by deleting the 300000 from the multi-text field and just pasting the first 1000 characters (from the long text of 300000 characters)
      5. When the notification is being sent the thread processing it will be take some time to calculate the difference of what change in the issue and will have the same stack trace as the one detailed in the description of the bug above.

       

      Expected Results

      The thread processing the notifications should process the differences faster or it should have a time out that can be configured by the administrator so when it takes a certain time to process a notification a template notification (such as "Too many things have change in this issue" or similar) is sent instead of spending a long time in calculating the changes.
      The administrator should be able to switch the time out on/off as well as being able to set the max time out interval.

       

      Actual Results

      The processing of the outgoing mail queue gets delayed, making the notifications to pile up in the queue and affecting the performance of it.

      The thread processing the notification will have the below stack trace:

       "Sending mailitem com.atlassian.jira.mail.IssueMailQueueItem@682a8028[issue=com.atlassian.jira.issue.IssueImpl@5122ec89[id=837382,summary=Lorem ipsum dolor sit amet, consectetur adipiscing elit ,key=PROJ-123,created=2017-01-06 19:27:14.107,updated=2017-08-18 05:50:47.763,assignee=user1(user1),reporter=user2(user2)],remoteUser=user3(user3),notificationType=Current_Assignee,eventTypeId=2,templateId=2]" #182 daemon prio=5 os_prio=0 tid=0x0000000058c36800 nid=0x1600 runnable [0x000000006631b000]
       java.lang.Thread.State: RUNNABLE
       at org.apache.commons.jrcs.diff.myers.MyersDiff.buildPath(MyersDiff.java:129)
       at org.apache.commons.jrcs.diff.myers.MyersDiff.diff(MyersDiff.java:93)
       at org.apache.commons.jrcs.diff.Diff.diff(Diff.java:197)
       at com.atlassian.diff.WordLevelDiffer.diffWords(WordLevelDiffer.java:101)
       at com.atlassian.diff.WordLevelDiffer.diffLine(WordLevelDiffer.java:91)
       at com.atlassian.diff.DiffViewBean.createWordLevelDiff(DiffViewBean.java:108)
       at com.atlassian.jira.mail.DiffUtils.diff(DiffUtils.java:19)
       at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
       at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
       at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
       at java.lang.reflect.Method.invoke(Method.java:498)
       at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.doInvoke(UberspectImpl.java:385)
       at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:374)
       at com.atlassian.velocity.htmlsafe.introspection.UnboxingMethod.invoke(UnboxingMethod.java:30)
      ...
       at com.atlassian.velocity.DefaultVelocityManager.writeEncodedBodyForContent(DefaultVelocityManager.java:86)
       at com.atlassian.jira.template.velocity.DefaultVelocityTemplatingEngine$DefaultRenderRequest.toWriterImpl(DefaultVelocityTemplatingEngine.java:129)
       at com.atlassian.jira.template.velocity.DefaultVelocityTemplatingEngine$DefaultRenderRequest.asPlainText(DefaultVelocityTemplatingEngine.java:108)
       at com.atlassian.jira.template.velocity.DefaultVelocityTemplatingEngine$DefaultRenderRequest$1.with(DefaultVelocityTemplatingEngine.java:92)
       at com.atlassian.jira.template.velocity.DefaultVelocityTemplatingEngine$DefaultRenderRequest$StringRepresentation.toString(DefaultVelocityTemplatingEngine.java:77)
       at com.atlassian.jira.template.velocity.DefaultVelocityTemplatingEngine$DefaultRenderRequest.asPlainText(DefaultVelocityTemplatingEngine.java:94)
       at com.atlassian.jira.mail.builder.EmailRenderer.renderEmailBody(EmailRenderer.java:103)
       at com.atlassian.jira.mail.builder.EmailRenderer.render(EmailRenderer.java:150)
       at com.atlassian.jira.mail.builder.EmailBuilder.renderNow(EmailBuilder.java:155)
       at com.atlassian.jira.mail.builder.EmailBuilder.renderNowAsQueueItem(EmailBuilder.java:145)
       at com.atlassian.jira.mail.MailingListCompiler$1.evaluateEmailForRecipient(MailingListCompiler.java:336)
       at com.atlassian.jira.mail.NotificationRecipientProcessor.evaluateEmails(NotificationRecipientProcessor.java:39)
       at com.atlassian.jira.mail.MailingListCompiler.evaluateEmails(MailingListCompiler.java:267)
      ...
       at com.atlassian.jira.mail.MailingListCompiler.prepareEmail(MailingListCompiler.java:170)
       at com.atlassian.jira.mail.MailingListCompiler.sendLists(MailingListCompiler.java:101)
       at com.atlassian.jira.mail.IssueMailQueueItem.send(IssueMailQueueItem.java:128)
       at com.atlassian.mail.queue.MailQueueImpl.sendBufferUnderLock(MailQueueImpl.java:103)
       at com.atlassian.mail.queue.MailQueueImpl.sendBuffer(MailQueueImpl.java:56)
       at com.atlassian.jira.mail.JiraMailQueue$1.apply(JiraMailQueue.java:51)
       at com.atlassian.jira.mail.JiraMailQueue$1.apply(JiraMailQueue.java:48)
       at com.atlassian.jira.util.velocity.DefaultVelocityRequestContextFactory.runWithStaticBaseUrl(DefaultVelocityRequestContextFactory.java:110)
       at com.atlassian.jira.util.DefaultBaseUrl.runWithStaticBaseUrl(DefaultBaseUrl.java:50)
       at com.atlassian.jira.mail.JiraMailQueue.sendBuffer(JiraMailQueue.java:48)
       at com.atlassian.jira.service.services.mail.MailQueueService.run(MailQueueService.java:21)
      ...
       at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:34)
       at java.lang.Thread.run(Thread.java:745)
       

       

      Workaround

      Flushing the queue can help to ease the problem as it add another thread to process notifications.
      Please note that you shouldn't flush the queue more then once, since it will use more Caesium (scheduler) threads.

      Additional Notes

      It is very important to note that the bigger the difference between the original text in the multi-line text field and what it was updated then the longer the diff libraries will take to process what it was change.
      For example comparing 1000 characters vs 300000 characters will take longer than comparing 240000 characters vs 300000 characters as the difference is smaller in the second comparison.
       

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              ecasuscelli Esteban Casuscelli
              Votes:
              13 Vote for this issue
              Watchers:
              23 Start watching this issue

              Dates

                Created:
                Updated: