-
Bug
-
Resolution: Fixed
-
Medium
-
2.0.1
I believe this might be a bug in the Service Desk Mail handler. If a user responds to a JIRA Service Desk email notification using Touchdown for Android that's connected through Exchange Active Sync (EAS), an exception will be recorded in the log files and the comment will not be added to the issue.
Note: Touchdown is a common Android application for interfacing with Microsoft Exchange for email, calendar, and contacts.
As a result, the issue is not updated with the comment and the assignee does not know the user commented. As you might imagine, that could lead to operational issues in responding to user requests in a timely manner.
To reproduce the issue, do the following:
1. Service Desk Customer submits a request through the customer portal
2. Service Desk Team adds comment to request
3. Service Desk Customer receives email notification containing comment and replies from an Android device using Touchdown that's connected through EAS
4. Log file will contain error message indicating the mail handler "failed due to binary incompatibilities"
I've included the error below. A non-trivial number of our users use Touchdown, so it's a significant issue for us. Our current workaround is to monitor all emails that come into the IMAP folder and verify that the comment was created in the issue.
2014-09-30 15:39:50,894 atlassian-scheduler-quartz1.clustered_Worker-2 ERROR ServiceRunner [atlassian.scheduler.core.JobLauncher] Scheduled job with ID 'com.atlassian.jira.service.JiraService:14200' failed due to binary incompatibilities java.lang.NoClassDefFoundError: org/apache/commons/lang3/StringUtils at com.atlassian.servicedesk.squalor.email.ServiceDeskMailUtils.stripQuotedLines(ServiceDeskMailUtils.java:182) at com.atlassian.servicedesk.internal.feature.incomingemail.IncomingEmailManager.addCommentFromEmail(IncomingEmailManager.scala:54) at com.atlassian.servicedesk.internal.feature.incomingemail.IncomingEmailService$$anonfun$addCommentIntoIssue$1$$anonfun$apply$8$$anonfun$apply$9$$anonfun$apply$10.apply(IncomingEmailService.scala:81) at com.atlassian.servicedesk.internal.feature.incomingemail.IncomingEmailService$$anonfun$addCommentIntoIssue$1$$anonfun$apply$8$$anonfun$apply$9$$anonfun$apply$10.apply(IncomingEmailService.scala:81) at com.atlassian.servicedesk.internal.feature.incomingemail.IncomingEmailService$$anonfun$com$$$$$b964c765cf7c52d46b2e78e0cf72fae$$$$inEmailContext$1$$anonfun$apply$15.apply(IncomingEmailService.scala:119) at com.atlassian.servicedesk.internal.utils.context.PortalContextUtil$.inPortalContextWithParam(PortalContextUtil.scala:32) at com.atlassian.servicedesk.internal.feature.incomingemail.IncomingEmailService$$anonfun$com$atlassian$servicedesk$internal$feature$incomingemail$IncomingEmailService$$inEmailContext$1.apply(IncomingEmailService.scala:118) at com.atlassian.servicedesk.internal.utils.context.AuthenticationContextUtil$.runAsUser(AuthenticationContextUtil.scala:15) at com.atlassian.servicedesk.internal.feature.incomingemail.IncomingEmailService.com$atlassian$servicedesk$internal$feature$incomingemail$IncomingEmailService$$inEmailContext(IncomingEmailService.scala:116) at com.atlassian.servicedesk.internal.feature.incomingemail.IncomingEmailService$$anonfun$addCommentIntoIssue$1$$anonfun$apply$8$$anonfun$apply$9.apply(IncomingEmailService.scala:79) at com.atlassian.servicedesk.internal.feature.incomingemail.IncomingEmailService$$anonfun$addCommentIntoIssue$1$$anonfun$apply$8$$anonfun$apply$9.apply(IncomingEmailService.scala:78) at scala.util.Either$RightProjection.flatMap(Either.scala:523) at com.atlassian.servicedesk.internal.feature.incomingemail.IncomingEmailService$$anonfun$addCommentIntoIssue$1$$anonfun$apply$8.apply(IncomingEmailService.scala:78) at com.atlassian.servicedesk.internal.feature.incomingemail.IncomingEmailService$$anonfun$addCommentIntoIssue$1$$anonfun$apply$8.apply(IncomingEmailService.scala:77) at scala.util.Either$RightProjection.flatMap(Either.scala:523) at com.atlassian.servicedesk.internal.feature.incomingemail.IncomingEmailService$$anonfun$addCommentIntoIssue$1.apply(IncomingEmailService.scala:77) at com.atlassian.servicedesk.internal.feature.incomingemail.IncomingEmailService$$anonfun$addCommentIntoIssue$1.apply(IncomingEmailService.scala:76) at scala.util.Either$RightProjection.flatMap(Either.scala:523) at com.atlassian.servicedesk.internal.feature.incomingemail.IncomingEmailService.addCommentIntoIssue(IncomingEmailService.scala:76) at com.atlassian.servicedesk.internal.feature.incomingemail.IncomingEmailService.processEmail(IncomingEmailService.scala:49) at com.atlassian.servicedesk.internal.email.SDMailHandler.processMessage(SDMailHandler.java:36) at com.atlassian.servicedesk.squalor.email.AbstractMailHandler.handleMessage(AbstractMailHandler.java:100) at com.atlassian.servicedesk.squalor.email.SDMessageProcessor.execute(SDMessageProcessor.java:77) at com.atlassian.servicedesk.squalor.email.ServiceDeskMailFetcherService.runImpl(ServiceDeskMailFetcherService.java:71) at com.atlassian.jira.service.services.file.AbstractMessageHandlingService.run(AbstractMessageHandlingService.java:261) at com.atlassian.jira.service.JiraServiceContainerImpl.run(JiraServiceContainerImpl.java:66) at com.atlassian.jira.service.ServiceRunner.runService(ServiceRunner.java:75) at com.atlassian.jira.service.ServiceRunner.runServiceId(ServiceRunner.java:53) at com.atlassian.jira.service.ServiceRunner.runJob(ServiceRunner.java:36) at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:135) at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:101) at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:80) at com.atlassian.scheduler.quartz1.Quartz1Job.execute(Quartz1Job.java:32) at org.quartz.core.JobRunShell.run(JobRunShell.java:223) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549) Caused by: java.lang.ClassNotFoundException: org.apache.commons.lang3.StringUtils at org.apache.felix.framework.ModuleImpl.findClassOrResourceByDelegation(ModuleImpl.java:772) at org.apache.felix.framework.ModuleImpl.access$200(ModuleImpl.java:73) at org.apache.felix.framework.ModuleImpl$ModuleClassLoader.loadClass(ModuleImpl.java:1690) at java.lang.ClassLoader.loadClass(ClassLoader.java:358) ... 35 more
| Form Name | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
[JSDSERVER-888] JIRA Service Desk mail handler fails to process email replies to email notifications in some circumstances
Odair Beraldo Nunes
added a comment - Guys, I need this fix desperately... this issue occurs in almost half of my clients domains
Can we have at least estimated date for when the Server-Next release will be available to download? Please?
Odair Beraldo Nunes
added a comment - Guys, I need this fix desperately... this issue occurs in almost half of my clients domains
Can we have at least estimated date for when the Server-Next release will be available to download? Please? 
J M
added a comment - This issue's original point, which I led off-course trying to investigate, is about errors with JIRA/Service Desk parsing e-mails sent by Touchdown clients which my understanding from our app people is an acknowledged bug by JIRA. We also are running into the issues described in JSD-827 and JSD-885 where issues aren't created when the receiver address is CCed or BCCed to the queue.
The final issue we thought JSD-904 described are actually due to how JIRA queries the IMAP connection with the UNSEEN search. We found a number of helpdesk people trying to interact with the mail via Outlook (or other mail client) which was resulting in the messages having the IMAP SEEN flag set which results in JIRA never seeing the mail.
J M
added a comment - This issue's original point, which I led off-course trying to investigate, is about errors with JIRA/Service Desk parsing e-mails sent by Touchdown clients which my understanding from our app people is an acknowledged bug by JIRA. We also are running into the issues described in JSD-827 and JSD-885 where issues aren't created when the receiver address is CCed or BCCed to the queue.
The final issue we thought JSD-904 described are actually due to how JIRA queries the IMAP connection with the UNSEEN search. We found a number of helpdesk people trying to interact with the mail via Outlook (or other mail client) which was resulting in the messages having the IMAP SEEN flag set which results in JIRA never seeing the mail. jasonmc What you are describing sounds like what is happening on https://jira.atlassian.com/browse/JSD-904 I think it would be worth watching that issue as well
J M
added a comment - I think we have a larger problem with Service Desk than just this corner case. We are reading in e-mail from an IMAP mailbox and messages appear to be randomly ignored/skipped, some with the error here and others silently. Is there documentation on the details of how JIRA uses IMAP and when it decides to take action on a message. Is it related only to the SEEN flag or does it keep some sort of Message-ID skiplist? Marking a message back to unread (i.e. remove \SEEN from the message) doesn't appear to cause JIRA to attempt to reprocess the message. The ONLY commonality we can find to all of the missed messages is they are multipart/* messages for Content-Type and all have some form of uuencoded binary inside. But there are no errors in the standard logging.
From jira-mail-incoming-log I see it doing an IMAP4 search of A4 SEARCH UNDELETED UNSEEN SINCE 3-Oct-2014 ALL but that doesn't seem to actually be all there is to it.
J M
added a comment - I think we have a larger problem with Service Desk than just this corner case. We are reading in e-mail from an IMAP mailbox and messages appear to be randomly ignored/skipped, some with the error here and others silently. Is there documentation on the details of how JIRA uses IMAP and when it decides to take action on a message. Is it related only to the SEEN flag or does it keep some sort of Message-ID skiplist? Marking a message back to unread (i.e. remove \SEEN from the message) doesn't appear to cause JIRA to attempt to reprocess the message. The ONLY commonality we can find to all of the missed messages is they are multipart/* messages for Content-Type and all have some form of uuencoded binary inside. But there are no errors in the standard logging.
From jira-mail-incoming-log I see it doing an IMAP4 search of A4 SEARCH UNDELETED UNSEEN SINCE 3-Oct-2014 ALL but that doesn't seem to actually be all there is to it. Thanks for the extended debugging you guys have done. You have done a fantastic job.
This has in fact ruled out my initial suspicion. I suggest we let future debugging go through the support process since you will get better service than via the public bug tracker where we cant get extended logs and so privately.
J M
added a comment - I am the aforementioned person who was looking at this on Tuesday. We are using the commons-lang3-3.1.jar that is provided in the WAR/EAR download for JIRA:
$ tar tvfz atlassian-jira-6.3.6-war.tar.gz | grep lang3 -rw-r--r-- bamboo-agent/bamboo-agent 315805 2014-09-15 20:25:12 atlassian-jira-6.3.6-war/webapp/WEB-INF/lib/commons-lang3-3.1.jar
I tried using a locally-built version of the latest commons-lang3 package from Apache (v3.2.2) and it made no difference. There is nothing we're altering about the classloader paths in Tomcat. Any JAR in the WEB-INF/lib structure loads based on other testing and historical use. I also tried adding it to the global JVM load location. I agree the NoClassDefFoundError usually means the JAR/class isn't available to the class loader, but I see nothing that would lead me to believe it's not present or being suppressed.
J M
added a comment - I am the aforementioned person who was looking at this on Tuesday. We are using the commons-lang3-3.1.jar that is provided in the WAR/EAR download for JIRA:
$ tar tvfz atlassian-jira-6.3.6-war.tar.gz | grep lang3
-rw-r--r-- bamboo-agent/bamboo-agent 315805 2014-09-15 20:25:12 atlassian-jira-6.3.6-war/webapp/WEB-INF/lib/commons-lang3-3.1.jar
I tried using a locally-built version of the latest commons-lang3 package from Apache (v3.2.2) and it made no difference. There is nothing we're altering about the classloader paths in Tomcat. Any JAR in the WEB-INF/lib structure loads based on other testing and historical use. I also tried adding it to the global JVM load location. I agree the NoClassDefFoundError usually means the JAR/class isn't available to the class loader, but I see nothing that would lead me to believe it's not present or being suppressed. Some additional analysis. We looked on the system to further investigate if it's a configuration issue on the server. We run JIRA inside of Tomcat 6 and checked to see where the StringUtils class was coming from.
[root@ourserver lib]# pwd
/usr/share/tomcat6/lib
[root@ourserver lib]# cd ..
[root@ourserver tomcat6]# ls
bin conf lib logs temp webapps work
[root@ourserver tomcat6]# cd webapps/jira/WEB-INF/lib/
[root@ourserver lib]# unzip -l commons-lang3-3.1.jar | grep StringUtil
40387 11-09-2011 22:58 org/apache/commons/lang3/StringUtils.class
2895 11-09-2011 22:58 org/apache/commons/lang3/StringUtils$InitStripAccents.class
2885 11-09-2011 22:58 org/apache/commons/lang3/RandomStringUtils.class
[root@ourserver lib]# pwd
/usr/share/tomcat6/webapps/jira/WEB-INF/lib
So it looks like the StringUtils class is being provided by a JIRA jar file. Would this have anything todo with the issue?
Do you guys need more examples of bad emails? I have like 30 raw emails that cause this issue.