Issue Summary

The Jira Service Management (JSM) mail handler might get stuck processing the same email when trying to convert it into a comment to be added to an existing ticket.
As a result, no new incoming emails are processed by the mail handler.

Expected Results

• The JSM Mail Handler should process any new incoming email.
• The Mail Processor job used by the JSM Mail Handler should not get stuck

Actual Results

• The Mail Processor job gets stuck processing the same incoming email, and does not move on to the next incoming email
  • When running the following SQL, we can see that the mail process job is stuck

    select * from rundetails where job_id like '%Processor%';
    
    1694420844712	1	1	NEW	1694420905548	jrey@atlassian.com	Email to test the JSM bug JSDSERVER-13967
    1694422468398	2	1	NEW	1694422468398	jrey@atlassian.com	Sending new email to see if it will be processed
    

• We can see in the thread dumps that the Mail Processor job is stuck performing some regex pattern:

  "Caesium-1-3" daemon prio=5 tid=0x00000000000002d0 nid=0 runnable 
     java.lang.Thread.State: RUNNABLE
  	at java.util.regex.Pattern$CharProperty.match(Pattern.java:3790)
  	at java.util.regex.Pattern$Curly.match0(Pattern.java:4274)
  	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
  	at java.util.regex.Pattern$Loop.matchInit(Pattern.java:4818)
  	at java.util.regex.Pattern$Prolog.match(Pattern.java:4755)
  	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
  	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
  	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
  	at java.util.regex.Pattern$Begin.match(Pattern.java:3539)
  	at java.util.regex.Matcher.search(Matcher.java:1248)
  	at java.util.regex.Matcher.find(Matcher.java:637)
  	at com.atlassian.pocketknife.internal.emailreply.matcher.basic.YahooSmbWroteOnDateBlockMatcher.isQuotedEmail(YahooSmbWroteOnDateBlockMatcher.java:25)
  	at com.atlassian.pocketknife.internal.emailreply.matcher.basic.DelegatingQuotedEmailMatcher.isQuotedEmail(DelegatingQuotedEmailMatcher.java:15)
  	at com.atlassian.pocketknife.internal.emailreply.DefaultEmailReplyCleaner.isQuotedBlock(DefaultEmailReplyCleaner.java:92)
  	at com.atlassian.pocketknife.internal.emailreply.DefaultEmailReplyCleaner.cleanQuotedEmailInternal(DefaultEmailReplyCleaner.java:79)
  	at com.atlassian.pocketknife.internal.emailreply.HtmlEmailReplyCleaner.cleanQuotedEmail(HtmlEmailReplyCleaner.java:138)
  	at com.atlassian.servicedesk.internal.email.replystripping.ServiceDeskEmailReplyCleaner.cleanQuotedEmail(ServiceDeskEmailReplyCleaner.java:161)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailParser.parse(IncomingEmailParser.java:108)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailManagerImpl.addServiceDeskCommentFromEmail(IncomingEmailManagerImpl.java:231)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.addServiceDeskCommentToIssue(IncomingEmailServiceImpl.java:332)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.lambda$addCommentToIssue$27(IncomingEmailServiceImpl.java:313)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl$$Lambda$13496/1164064939.apply(Unknown Source)
  	at io.atlassian.fugue.Option$Some.fold(Option.java:444)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.addCommentToIssue(IncomingEmailServiceImpl.java:310)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.lambda$null$18(IncomingEmailServiceImpl.java:208)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl$$Lambda$11743/2040580230.apply(Unknown Source)
  	at io.atlassian.fugue.Option$Some.fold(Option.java:444)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.lambda$createNewOrCommentOnIssueWithMailChannelAsEmailSender$19(IncomingEmailServiceImpl.java:198)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl$$Lambda$11732/1953097413.call(Unknown Source)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl$$Lambda$11735/531084411.get(Unknown Source)
  	at io.atlassian.fugue.Option$None.fold(Option.java:393)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.createNewOrCommentOnIssueWithMailChannelAsEmailSender(IncomingEmailServiceImpl.java:223)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.lambda$null$14(IncomingEmailServiceImpl.java:185)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl$$Lambda$11728/820346052.get(Unknown Source)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl$$Lambda$11731/1770925126.call(Unknown Source)
  	at com.atlassian.servicedesk.internal.api.util.context.ReentrantThreadLocalBasedCodeContext.rteInvoke(ReentrantThreadLocalBasedCodeContext.java:136)
  	at com.atlassian.servicedesk.internal.api.util.context.ReentrantThreadLocalBasedCodeContext.runInContext(ReentrantThreadLocalBasedCodeContext.java:54)
  	at com.atlassian.servicedesk.internal.utils.context.CustomerContextServiceImpl.runInCustomerContext(CustomerContextServiceImpl.java:37)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.inCustomerContext(IncomingEmailServiceImpl.java:509)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.lambda$inEmailContext$44(IncomingEmailServiceImpl.java:502)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl$$Lambda$11730/162278593.call(Unknown Source)
  	at com.atlassian.pocketknife.internal.util.runner.AuthenticationContextUtilImpl.runAs(AuthenticationContextUtilImpl.java:53)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.inEmailContext(IncomingEmailServiceImpl.java:504)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.lambda$doCreateNewOrCommentOnIssueWithMailChannel$15(IncomingEmailServiceImpl.java:185)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl$$Lambda$11723/189973775.apply(Unknown Source)
  	at io.atlassian.fugue.Either$Right.fold(Either.java:641)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.doCreateNewOrCommentOnIssueWithMailChannel(IncomingEmailServiceImpl.java:183)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.lambda$createNewOrCommentOnIssueWithMailChannel$6(IncomingEmailServiceImpl.java:153)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl$$Lambda$11720/1542200341.apply(Unknown Source)
  	at com.atlassian.pocketknife.step.EitherStep2.lambda$null$0(EitherStep2.java:20)
  	at com.atlassian.pocketknife.step.EitherStep2$$Lambda$5808/89363710.apply(Unknown Source)
  	at io.atlassian.fugue.Either$RightProjection.flatMap(Either.java:937)
  	at io.atlassian.fugue.Either.flatMap(Either.java:231)
  	at com.atlassian.pocketknife.step.EitherStep2.lambda$then$1(EitherStep2.java:20)
  	at com.atlassian.pocketknife.step.EitherStep2$$Lambda$5807/1542923396.apply(Unknown Source)
  	at io.atlassian.fugue.Either$RightProjection.flatMap(Either.java:937)
  	at io.atlassian.fugue.Either.flatMap(Either.java:231)
  	at com.atlassian.pocketknife.step.EitherStep2.then(EitherStep2.java:20)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.createNewOrCommentOnIssueWithMailChannel(IncomingEmailServiceImpl.java:153)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.lambda$processEmailWithMailChannel$1(IncomingEmailServiceImpl.java:141)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl$$Lambda$11585/163139394.apply(Unknown Source)
  	at com.atlassian.pocketknife.step.EitherStep1$$Lambda$5476/1873075899.apply(Unknown Source)
  	at io.atlassian.fugue.Either$RightProjection.flatMap(Either.java:937)
  	at io.atlassian.fugue.Either.flatMap(Either.java:231)
  	at com.atlassian.pocketknife.step.EitherStep1.then(EitherStep1.java:18)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.processEmailWithMailChannel(IncomingEmailServiceImpl.java:141)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.emailplatform.handler.SDMailHandler.processMessage(SDMailHandler.java:75)
  	at com.atlassian.servicedesk.internal.feature.emailchannel.emailplatform.handler.SDMailHandler.handleReplyMessage(SDMailHandler.java:57)
  	at com.atlassian.jira.internal.mail.processor.feature.processor.MailHandlerWorker.handle(MailHandlerWorker.java:56)
  	at com.atlassian.jira.internal.mail.processor.feature.processor.MailProcessorWorker.processMailItem(MailProcessorWorker.java:121)
  	at com.atlassian.jira.internal.mail.processor.feature.processor.MailProcessorWorker.lambda$process$0(MailProcessorWorker.java:96)
  	at com.atlassian.jira.internal.mail.processor.feature.processor.MailProcessorWorker$$Lambda$11499/2069769991.apply(Unknown Source)
  	at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl.lambda$execute$0(DatabaseAccessorImpl.java:69)
  	at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl$$Lambda$4721/260063885.execute(Unknown Source)
  	at com.atlassian.sal.core.rdbms.DefaultTransactionalExecutor.executeInternal(DefaultTransactionalExecutor.java:109)
  	at com.atlassian.sal.core.rdbms.DefaultTransactionalExecutor.lambda$execute$0(DefaultTransactionalExecutor.java:64)
  	at com.atlassian.sal.core.rdbms.DefaultTransactionalExecutor$$Lambda$1449/1232156179.execute(Unknown Source)
  	at com.atlassian.sal.jira.rdbms.JiraHostConnectionAccessor.lambda$borrowConnectionAndExecute$0(JiraHostConnectionAccessor.java:95)
  	at com.atlassian.sal.jira.rdbms.JiraHostConnectionAccessor$$Lambda$4082/691077169.run(Unknown Source)
  	at com.atlassian.jira.database.DatabaseAccessorImpl.executeQuery(DatabaseAccessorImpl.java:74)
  	at sun.reflect.GeneratedMethodAccessor415.invoke(Unknown Source)
  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.lang.reflect.Method.invoke(Method.java:498)
  	at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
  	at com.sun.proxy.$Proxy284.executeQuery(Unknown Source)
  	at sun.reflect.GeneratedMethodAccessor415.invoke(Unknown Source)
  	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  	at java.lang.reflect.Method.invoke(Method.java:498)
  	at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(HostComponentFactoryBean.java:130)
  	at com.sun.proxy.$Proxy284.executeQuery(Unknown Source)
  	at com.atlassian.sal.jira.rdbms.JiraHostConnectionAccessor.borrowConnectionAndExecute(JiraHostConnectionAccessor.java:84)
  	at com.atlassian.sal.jira.rdbms.JiraHostConnectionAccessor.execute(JiraHostConnectionAccessor.java:48)
  	at com.atlassian.sal.core.rdbms.DefaultTransactionalExecutor.execute(DefaultTransactionalExecutor.java:61)
  	at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl.execute(DatabaseAccessorImpl.java:67)
  	at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl.runInNewTransaction(DatabaseAccessorImpl.java:38)
  	at com.atlassian.jira.internal.mail.processor.feature.processor.MailProcessorWorker.process(MailProcessorWorker.java:93)
  	at com.atlassian.jira.internal.mail.processor.feature.processor.MailProcessorWorker.processAllValidMailChannels(MailProcessorWorker.java:143)
  	at com.atlassian.jira.internal.mail.processor.feature.processor.MailProcessorService.run(MailProcessorService.java:25)
  	at com.atlassian.jira.internal.mail.processor.services.MailProcessorExecutor.run(MailProcessorExecutor.java:29)
  	at com.atlassian.jira.internal.mail.processor.services.AbstractMailExecutor.execute(AbstractMailExecutor.java:45)
  	at com.atlassian.jira.internal.mail.processor.services.MailJobRunner.runJob(MailJobRunner.java:35)
  	at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
  	at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
  	at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
  	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
  	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
  	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
  	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
  	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$$Lambda$4944/1221236421.accept(Unknown Source)
  	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
  	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
  	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
  	at java.lang.Thread.run(Thread.java:750)
  
     Locked ownable synchronizers:
  	- None
  

Workaround

• Get the first affected mail item using the below-mentioned query:

   select * from AO_2C4E5C_MAILITEM where status = "NEW" order by ASC 

• Get the value from ID column from the first row and use it in the next query:

  select * from AO_2C4E5C_MAILITEMCHUNK where MAIL_ITEM_ID=<ID from previous query>

• Get the email source in AO_2C4E5C_MAILITEMCHUNK using that ID, and base 64 decode it. This should be the problematic email that is making the mail processor stuck.

How to resolve this issue:

• Mark the problematic email as processed by executing below mentioned command. Replace <ID> with the ID identified in the previous steps

  UPDATE "AO_2C4E5C_MAILITEM" SET "STATUS" = 'PROCESSED' where "ID" = <ID>;

• Restart the Jira nodes