[JSDSERVER-13967] Jira Service Management Mail Process can get stuck processing the same email

Type: Bug
Resolution: Fixed
Priority: High
Fix Version/s: 5.13.0, 4.20.30, 5.4.15, 5.12.2
Affects Version/s: 5.4.7, 5.10.0, 5.11.0
Component/s: Email - Incoming
Labels:
- architectural

Support reference count:
1
Symptom Severity:
Severity 2 - Major
Bug Fix Policy:
View Atlassian Server bug fix policy

Issue Summary

The Jira Service Management (JSM) mail handler might get stuck processing the same email when trying to convert it into a comment to be added to an existing ticket.
As a result, no new incoming emails are processed by the mail handler.

Expected Results

The JSM Mail Handler should process any new incoming email.
The Mail Processor job used by the JSM Mail Handler should not get stuck

Actual Results

The Mail Processor job gets stuck processing the same incoming email, and does not move on to the next incoming email

When running the following SQL, we can see that the mail process job is stuck

select * from rundetails where job_id like '%Processor%';

1694420844712	1	1	NEW	1694420905548	jrey@atlassian.com	Email to test the JSM bug JSDSERVER-13967
1694422468398	2	1	NEW	1694422468398	jrey@atlassian.com	Sending new email to see if it will be processed

We can see in the thread dumps that the Mail Processor job is stuck performing some regex pattern:

"Caesium-1-3" daemon prio=5 tid=0x00000000000002d0 nid=0 runnable 
   java.lang.Thread.State: RUNNABLE
	at java.util.regex.Pattern$CharProperty.match(Pattern.java:3790)
	at java.util.regex.Pattern$Curly.match0(Pattern.java:4274)
	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
	at java.util.regex.Pattern$GroupHead.match(Pattern.java:4672)
	at java.util.regex.Pattern$Loop.matchInit(Pattern.java:4818)
	at java.util.regex.Pattern$Prolog.match(Pattern.java:4755)
	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
	at java.util.regex.Pattern$Curly.match0(Pattern.java:4286)
	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
	at java.util.regex.Pattern$Curly.match0(Pattern.java:4293)
	at java.util.regex.Pattern$Curly.match(Pattern.java:4248)
	at java.util.regex.Pattern$Begin.match(Pattern.java:3539)
	at java.util.regex.Matcher.search(Matcher.java:1248)
	at java.util.regex.Matcher.find(Matcher.java:637)
	at com.atlassian.pocketknife.internal.emailreply.matcher.basic.YahooSmbWroteOnDateBlockMatcher.isQuotedEmail(YahooSmbWroteOnDateBlockMatcher.java:25)
	at com.atlassian.pocketknife.internal.emailreply.matcher.basic.DelegatingQuotedEmailMatcher.isQuotedEmail(DelegatingQuotedEmailMatcher.java:15)
	at com.atlassian.pocketknife.internal.emailreply.DefaultEmailReplyCleaner.isQuotedBlock(DefaultEmailReplyCleaner.java:92)
	at com.atlassian.pocketknife.internal.emailreply.DefaultEmailReplyCleaner.cleanQuotedEmailInternal(DefaultEmailReplyCleaner.java:79)
	at com.atlassian.pocketknife.internal.emailreply.HtmlEmailReplyCleaner.cleanQuotedEmail(HtmlEmailReplyCleaner.java:138)
	at com.atlassian.servicedesk.internal.email.replystripping.ServiceDeskEmailReplyCleaner.cleanQuotedEmail(ServiceDeskEmailReplyCleaner.java:161)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailParser.parse(IncomingEmailParser.java:108)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailManagerImpl.addServiceDeskCommentFromEmail(IncomingEmailManagerImpl.java:231)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.addServiceDeskCommentToIssue(IncomingEmailServiceImpl.java:332)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.lambda$addCommentToIssue$27(IncomingEmailServiceImpl.java:313)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl$$Lambda$13496/1164064939.apply(Unknown Source)
	at io.atlassian.fugue.Option$Some.fold(Option.java:444)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.addCommentToIssue(IncomingEmailServiceImpl.java:310)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.lambda$null$18(IncomingEmailServiceImpl.java:208)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl$$Lambda$11743/2040580230.apply(Unknown Source)
	at io.atlassian.fugue.Option$Some.fold(Option.java:444)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.lambda$createNewOrCommentOnIssueWithMailChannelAsEmailSender$19(IncomingEmailServiceImpl.java:198)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl$$Lambda$11732/1953097413.call(Unknown Source)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl$$Lambda$11735/531084411.get(Unknown Source)
	at io.atlassian.fugue.Option$None.fold(Option.java:393)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.createNewOrCommentOnIssueWithMailChannelAsEmailSender(IncomingEmailServiceImpl.java:223)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.lambda$null$14(IncomingEmailServiceImpl.java:185)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl$$Lambda$11728/820346052.get(Unknown Source)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl$$Lambda$11731/1770925126.call(Unknown Source)
	at com.atlassian.servicedesk.internal.api.util.context.ReentrantThreadLocalBasedCodeContext.rteInvoke(ReentrantThreadLocalBasedCodeContext.java:136)
	at com.atlassian.servicedesk.internal.api.util.context.ReentrantThreadLocalBasedCodeContext.runInContext(ReentrantThreadLocalBasedCodeContext.java:54)
	at com.atlassian.servicedesk.internal.utils.context.CustomerContextServiceImpl.runInCustomerContext(CustomerContextServiceImpl.java:37)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.inCustomerContext(IncomingEmailServiceImpl.java:509)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.lambda$inEmailContext$44(IncomingEmailServiceImpl.java:502)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl$$Lambda$11730/162278593.call(Unknown Source)
	at com.atlassian.pocketknife.internal.util.runner.AuthenticationContextUtilImpl.runAs(AuthenticationContextUtilImpl.java:53)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.inEmailContext(IncomingEmailServiceImpl.java:504)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.lambda$doCreateNewOrCommentOnIssueWithMailChannel$15(IncomingEmailServiceImpl.java:185)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl$$Lambda$11723/189973775.apply(Unknown Source)
	at io.atlassian.fugue.Either$Right.fold(Either.java:641)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.doCreateNewOrCommentOnIssueWithMailChannel(IncomingEmailServiceImpl.java:183)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.lambda$createNewOrCommentOnIssueWithMailChannel$6(IncomingEmailServiceImpl.java:153)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl$$Lambda$11720/1542200341.apply(Unknown Source)
	at com.atlassian.pocketknife.step.EitherStep2.lambda$null$0(EitherStep2.java:20)
	at com.atlassian.pocketknife.step.EitherStep2$$Lambda$5808/89363710.apply(Unknown Source)
	at io.atlassian.fugue.Either$RightProjection.flatMap(Either.java:937)
	at io.atlassian.fugue.Either.flatMap(Either.java:231)
	at com.atlassian.pocketknife.step.EitherStep2.lambda$then$1(EitherStep2.java:20)
	at com.atlassian.pocketknife.step.EitherStep2$$Lambda$5807/1542923396.apply(Unknown Source)
	at io.atlassian.fugue.Either$RightProjection.flatMap(Either.java:937)
	at io.atlassian.fugue.Either.flatMap(Either.java:231)
	at com.atlassian.pocketknife.step.EitherStep2.then(EitherStep2.java:20)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.createNewOrCommentOnIssueWithMailChannel(IncomingEmailServiceImpl.java:153)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.lambda$processEmailWithMailChannel$1(IncomingEmailServiceImpl.java:141)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl$$Lambda$11585/163139394.apply(Unknown Source)
	at com.atlassian.pocketknife.step.EitherStep1$$Lambda$5476/1873075899.apply(Unknown Source)
	at io.atlassian.fugue.Either$RightProjection.flatMap(Either.java:937)
	at io.atlassian.fugue.Either.flatMap(Either.java:231)
	at com.atlassian.pocketknife.step.EitherStep1.then(EitherStep1.java:18)
	at com.atlassian.servicedesk.internal.feature.emailchannel.IncomingEmailServiceImpl.processEmailWithMailChannel(IncomingEmailServiceImpl.java:141)
	at com.atlassian.servicedesk.internal.feature.emailchannel.emailplatform.handler.SDMailHandler.processMessage(SDMailHandler.java:75)
	at com.atlassian.servicedesk.internal.feature.emailchannel.emailplatform.handler.SDMailHandler.handleReplyMessage(SDMailHandler.java:57)
	at com.atlassian.jira.internal.mail.processor.feature.processor.MailHandlerWorker.handle(MailHandlerWorker.java:56)
	at com.atlassian.jira.internal.mail.processor.feature.processor.MailProcessorWorker.processMailItem(MailProcessorWorker.java:121)
	at com.atlassian.jira.internal.mail.processor.feature.processor.MailProcessorWorker.lambda$process$0(MailProcessorWorker.java:96)
	at com.atlassian.jira.internal.mail.processor.feature.processor.MailProcessorWorker$$Lambda$11499/2069769991.apply(Unknown Source)
	at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl.lambda$execute$0(DatabaseAccessorImpl.java:69)
	at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl$$Lambda$4721/260063885.execute(Unknown Source)
	at com.atlassian.sal.core.rdbms.DefaultTransactionalExecutor.executeInternal(DefaultTransactionalExecutor.java:109)
	at com.atlassian.sal.core.rdbms.DefaultTransactionalExecutor.lambda$execute$0(DefaultTransactionalExecutor.java:64)
	at com.atlassian.sal.core.rdbms.DefaultTransactionalExecutor$$Lambda$1449/1232156179.execute(Unknown Source)
	at com.atlassian.sal.jira.rdbms.JiraHostConnectionAccessor.lambda$borrowConnectionAndExecute$0(JiraHostConnectionAccessor.java:95)
	at com.atlassian.sal.jira.rdbms.JiraHostConnectionAccessor$$Lambda$4082/691077169.run(Unknown Source)
	at com.atlassian.jira.database.DatabaseAccessorImpl.executeQuery(DatabaseAccessorImpl.java:74)
	at sun.reflect.GeneratedMethodAccessor415.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
	at com.sun.proxy.$Proxy284.executeQuery(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor415.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(HostComponentFactoryBean.java:130)
	at com.sun.proxy.$Proxy284.executeQuery(Unknown Source)
	at com.atlassian.sal.jira.rdbms.JiraHostConnectionAccessor.borrowConnectionAndExecute(JiraHostConnectionAccessor.java:84)
	at com.atlassian.sal.jira.rdbms.JiraHostConnectionAccessor.execute(JiraHostConnectionAccessor.java:48)
	at com.atlassian.sal.core.rdbms.DefaultTransactionalExecutor.execute(DefaultTransactionalExecutor.java:61)
	at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl.execute(DatabaseAccessorImpl.java:67)
	at com.atlassian.pocketknife.internal.querydsl.DatabaseAccessorImpl.runInNewTransaction(DatabaseAccessorImpl.java:38)
	at com.atlassian.jira.internal.mail.processor.feature.processor.MailProcessorWorker.process(MailProcessorWorker.java:93)
	at com.atlassian.jira.internal.mail.processor.feature.processor.MailProcessorWorker.processAllValidMailChannels(MailProcessorWorker.java:143)
	at com.atlassian.jira.internal.mail.processor.feature.processor.MailProcessorService.run(MailProcessorService.java:25)
	at com.atlassian.jira.internal.mail.processor.services.MailProcessorExecutor.run(MailProcessorExecutor.java:29)
	at com.atlassian.jira.internal.mail.processor.services.AbstractMailExecutor.execute(AbstractMailExecutor.java:45)
	at com.atlassian.jira.internal.mail.processor.services.MailJobRunner.runJob(MailJobRunner.java:35)
	at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:134)
	at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:106)
	at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:90)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:435)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:430)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:454)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:382)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$$Lambda$4944/1221236421.accept(Unknown Source)
	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:66)
	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:60)
	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:35)
	at java.lang.Thread.run(Thread.java:750)

   Locked ownable synchronizers:
	- None

Workaround

Get the first affected mail item using the below-mentioned query:

 select * from AO_2C4E5C_MAILITEM where status = "NEW" order by ASC

Get the value from ID column from the first row and use it in the next query:

select * from AO_2C4E5C_MAILITEMCHUNK where MAIL_ITEM_ID=<ID from previous query>

Get the email source in AO_2C4E5C_MAILITEMCHUNK using that ID, and base 64 decode it. This should be the problematic email that is making the mail processor stuck.

How to resolve this issue:

Mark the problematic email as processed by executing below mentioned command. Replace <ID> with the ID identified in the previous steps
```
UPDATE "AO_2C4E5C_MAILITEM" SET "STATUS" = 'PROCESSED' where "ID" = <ID>;
```
Restart the Jira nodes

mentioned in: Page Failed to load; Page Failed to load

Form Name

Satej Mirpagar made changes - 02/Jan/2024 10:23 AM

Resolution		New: Fixed [ 1 ]
Status	Original: Waiting for Release [ 12075 ]	New: Closed [ 6 ]

Tripta Kaur made changes - 28/Dec/2023 12:46 AM

Remote Link

New: This issue links to "Page (Confluence)" [ 850812 ]

Tripta Kaur made changes - 07/Dec/2023 6:05 AM

Assignee

New: Tripta Kaur [ 399904f8a539 ]

Tripta Kaur made changes - 07/Dec/2023 3:41 AM

Status

Original: In Progress [ 3 ]

New: Waiting for Release [ 12075 ]

Tripta Kaur made changes - 07/Dec/2023 3:41 AM

Fix Version/s		New: 5.12.2 [ 106520 ]
Fix Version/s		New: 5.4.15 [ 106514 ]
Fix Version/s		New: 4.20.30 [ 106365 ]
Fix Version/s		New: 5.13.0 [ 105935 ]

Alex Cooksey made changes - 01/Dec/2023 2:46 AM

Remote Link

New: This issue links to "Page (Confluence)" [ 843299 ]

Tripta Kaur made changes - 21/Nov/2023 2:31 AM

Status

Original: Waiting for Release [ 12075 ]

New: In Progress [ 3 ]

server-jirasync-bot made changes - 21/Nov/2023 2:06 AM

Status

Original: In Progress [ 3 ]

New: Waiting for Release [ 12075 ]

server-jirasync-bot made changes - 21/Nov/2023 2:06 AM

Status

Original: Short Term Backlog [ 12074 ]

New: In Progress [ 3 ]

Alex Cooksey made changes - 26/Sep/2023 12:12 AM

Labels

New: architectural

Assignee:: Tripta Kaur

Reporter:: Vedika Tambolkar

Affected customers:: 1 This affects my team

Watchers:: 5 Start watching this issue

Created:: 02/Aug/2023 11:38 AM

Updated:: 02/Jan/2024 10:23 AM

Resolved:: 02/Jan/2024 10:23 AM

Details

Description

Issue Summary

Expected Results

Actual Results

Workaround

Attachments

Issue Links

Forms

Activity

People

Dates

Backbone Issue Sync