Uploaded image for project: 'Jira Service Management Server and Data Center'
  1. Jira Service Management Server and Data Center
  2. JSDSERVER-6099

Emails with Large Size Attachments Fail/Are slow To Generate Ticket in IMAP Mailbox

    XMLWordPrintable

Details

    Description

      Summary

      Emails sent to an IMAP custom Service Desk inbox with attachments bigger than 2Mb will not create a ticket in their JSD project. The email is read in the mailbox though but there is no record of JSD ever processing it. The issue is reproducible by the customer as the email is currently sitting in their inbox as read, and has been marked as unread several times. The email gets then marked as read with no success in creating the ticket in JSD.

      Steps to Reproduce

      1. Setup custom secure IMAP mail server
      2. Receive an email with attachment of around 2Mb or larger
      3. Notice the email will be read, but no record is logged in Jira or the Attachment takes 15 mins to be processed.

      NOTE: This could not be reproduced in our Atlassian test environment.

      Expected Results

      1. Email is set with attachment
      2. Ticket is created and a log created in Jira

      Actual Results

      Email is marked as read but JSD ticket does not get generated and the following stack trace is found in the logs:

      2018-10-17 08:05:21,044 ERROR [] Caesium-1-4 ServiceRunner     Error while serializing message
      com.sun.mail.util.FolderClosedIOException
              at com.sun.mail.imap.IMAPInputStream.forceCheckExpunged(IMAPInputStream.java:107) [javax.mail-1.6.0.jar:1.6.0]
              at com.sun.mail.imap.IMAPInputStream.fill(IMAPInputStream.java:158) [javax.mail-1.6.0.jar:1.6.0]
              at com.sun.mail.imap.IMAPInputStream.read(IMAPInputStream.java:218) [javax.mail-1.6.0.jar:1.6.0]
              at com.sun.mail.imap.IMAPInputStream.read(IMAPInputStream.java:244) [javax.mail-1.6.0.jar:1.6.0]
              at com.sun.mail.imap.IMAPMessage.writeTo(IMAPMessage.java:849) [javax.mail-1.6.0.jar:1.6.0]
              at com.atlassian.jira.internal.mail.processor.util.MimeMessageBase64ChunkStream.fullyLoadMessageToMemory(MimeMessageBase64ChunkStream.java:32) [?:?]
              at com.atlassian.jira.internal.mail.processor.util.MimeMessageBase64ChunkStream.<init>(MimeMessageBase64ChunkStream.java:22) [?:?]
              at com.atlassian.jira.internal.mail.processor.feature.puller.mailitem.MailItemStore.createMailItemChunk(MailItemStore.java:239) [?:?]
              at com.atlassian.jira.internal.mail.processor.feature.puller.mailitem.MailItemStore.lambda$null$4(MailItemStore.java:214) [?:?]
              at io.atlassian.fugue.Option.map(Option.java:259) [fugue-4.5.0.jar:?]
              at com.atlassian.jira.internal.mail.processor.feature.puller.mailitem.MailItemStore.lambda$createMailItem$5(MailItemStore.java:212) [?:?]
              at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl.lambda$execute$0(DatabaseAccessorImpl.java:69) [?:?]
              at com.atlassian.sal.core.rdbms.DefaultTransactionalExecutor.executeInternal(DefaultTransactionalExecutor.java:86) [?:?]
              at com.atlassian.sal.core.rdbms.DefaultTransactionalExecutor.lambda$execute$0(DefaultTransactionalExecutor.java:42) [?:?]
              at com.atlassian.jira.database.DatabaseAccessorImpl.runInManagedTransaction(DatabaseAccessorImpl.java:125) [classes/:?]
              at sun.reflect.GeneratedMethodAccessor234.invoke(Unknown Source) [?:?]
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [?:1.8.0_102]
              at java.lang.reflect.Method.invoke(Method.java:498) [?:1.8.0_102]
              at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26) [atlassian-plugins-core-4.5.1.jar:?]
              at com.sun.proxy.$Proxy114.runInManagedTransaction(Unknown Source) [?:?]
              at sun.reflect.GeneratedMethodAccessor234.invoke(Unknown Source) [?:?]
              at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [?:1.8.0_102]
              at java.lang.reflect.Method.invoke(Method.java:498) [?:1.8.0_102]
              at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(HostComponentFactoryBean.java:136) [atlassian-plugins-osgi-bridge-4.5.1.jar:?]
              at com.sun.proxy.$Proxy114.runInManagedTransaction(Unknown Source) [?:?]
              at com.atlassian.sal.jira.rdbms.JiraHostConnectionAccessor.runInStartedOrExistingTransaction(JiraHostConnectionAccessor.java:130) [jira-sal-plugin-7.11.2_1533715698000.jar:?]
              at com.atlassian.sal.jira.rdbms.JiraHostConnectionAccessor.execute(JiraHostConnectionAccessor.java:60) [jira-sal-plugin-7.11.2_1533715698000.jar:?]
              at com.atlassian.sal.core.rdbms.DefaultTransactionalExecutor.execute(DefaultTransactionalExecutor.java:39) [jira-sal-plugin-7.11.2_1533715698000.jar:?]
              at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl.execute(DatabaseAccessorImpl.java:67) [jira-email-processor-plugin-2.11.0.jar:?]
              at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl.runInTransaction(DatabaseAccessorImpl.java:43) [jira-email-processor-plugin-2.11.0.jar:?]
              at com.atlassian.pocketknife.api.querydsl.DatabaseAccessor.runInTransaction(DatabaseAccessor.java:134) [jira-email-processor-plugin-2.11.0.jar:?]
              at com.atlassian.jira.internal.mail.processor.querydsl.QueryDslHelper.runInTransaction(QueryDslHelper.java:53) [jira-email-processor-plugin-2.11.0.jar:?]
              at com.atlassian.jira.internal.mail.processor.feature.puller.mailitem.MailItemStore.createMailItem(MailItemStore.java:208) [jira-email-processor-plugin-2.11.0.jar:?]
              at com.atlassian.jira.internal.mail.processor.feature.puller.mailitem.MailItemManager.createMailItemFromMessage(MailItemManager.java:38) [jira-email-processor-plugin-2.11.0.jar:?]
              at com.atlassian.jira.internal.mail.processor.feature.puller.MailPullerWorker.persistMailMessage(MailPullerWorker.java:209) [jira-email-processor-plugin-2.11.0.jar:?]
              at com.atlassian.jira.internal.mail.processor.feature.puller.MailPullerWorker.pullEmailForConnection(MailPullerWorker.java:113) [jira-email-processor-plugin-2.11.0.jar:?]
              at com.atlassian.jira.internal.mail.processor.feature.puller.MailPullerWorker.pullMailFromAllValidChannels(MailPullerWorker.java:76) [jira-email-processor-plugin-2.11.0.jar:?]
              at com.atlassian.jira.internal.mail.processor.feature.puller.MailPullerService.run(MailPullerService.java:33) [jira-email-processor-plugin-2.11.0.jar:?]
              at com.atlassian.jira.internal.mail.processor.services.MailPullerExecutor.run(MailPullerExecutor.java:29) [jira-email-processor-plugin-2.11.0.jar:?]
              at com.atlassian.jira.internal.mail.processor.services.AbstractMailExecutor.execute(AbstractMailExecutor.java:45) [jira-email-processor-plugin-2.11.0.jar:?]
              at com.atlassian.jira.internal.mail.processor.services.MailJobRunner.runJob(MailJobRunner.java:35) [jira-email-processor-plugin-2.11.0.jar:?]
              at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:153) [atlassian-scheduler-core-1.7.0.jar:?]
              at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:118) [atlassian-scheduler-core-1.7.0.jar:?]
              at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:97) [atlassian-scheduler-core-1.7.0.jar:?]
              at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:443) [atlassian-scheduler-caesium-1.3.3.jar:?]
              at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:438) [atlassian-scheduler-caesium-1.3.3.jar:?]
              at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:462) [atlassian-scheduler-caesium-1.3.3.jar:?]
              at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:390) [atlassian-scheduler-caesium-1.3.3.jar:?]
              at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:285) [atlassian-scheduler-caesium-1.3.3.jar:?]
              at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:282) [atlassian-scheduler-caesium-1.3.3.jar:?]
              at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:65) [atlassian-scheduler-caesium-1.3.3.jar:?]
              at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:59) [atlassian-scheduler-caesium-1.3.3.jar:?]
              at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:34) [atlassian-scheduler-caesium-1.3.3.jar:?]
              at java.lang.Thread.run(Thread.java:745) [?:1.8.0_102]
      2018-10-17 08:05:21,106 DEBUG [] Caesium-1-4 ServiceRunner     The result of deletion 0
      2018-10-17 08:05:21,106 DEBUG [] Caesium-1-4 ServiceRunner     The result of deletion 1
      

      This always happens for all the emails that are slow or fail to be processed.

      Notes

      • By looking at the connectivity logs for that inbox, we can see several of [AUTHENTICATIONFAILED] Invalid credentials (Failure) errors in the last month.
      • Forward the email directly to the default Jira inbox, a ticket was created

      Workaround

      Workaround 1

      There's been some evidence that moving to POP mitigates the bug

      Workaround 2

      There's been some evidence that adding the 2 following JVM parameters and re-starting the Jira application also helps mitigate the bug:

      -Dmail.imap.fetchsize=1048576
      -Dmail.imaps.fetchsize=1048576
      

      These settings will increase the Java Mail Prefetch size from 16kb to 1mb, in order to limit the number of requests sent from the Service Desk mail handler to the network/

      The steps are:

      • Stop the Jira application (stop all the Jira nodes if using Jira Data Center)
      • Add the following JVM parameters as per this documentation
        -Dmail.imap.fetchsize=1048576
        -Dmail.imaps.fetchsize=1048576
        
      • Backup the Jira Database
      • Run the following DELETE query in the Jira Database (this step will help reset the Service Desk Mail Puller job, in case it was stuck. Note the row corresponding to this job will be automatically re-created in the rundetails table after a Jira re-start)
        delete from rundetails where job_id = 'com.atlassian.jira.internal.mail.services.MailPullerJobRunner';
        
      • Start the Jira application (or start all the Jira nodes id using Jira Data Center)

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              nsilvaratnam Niro (Inactive)
              Votes:
              12 Vote for this issue
              Watchers:
              18 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Backbone Issue Sync