Uploaded image for project: 'Jira Service Management Data Center'
  1. Jira Service Management Data Center
  2. JSDSERVER-4550

Strip quotes not working on incoming mails containing text formatting

      NOTE: This bug report is for JIRA Service Desk Server. Using JIRA Service Desk Cloud? See the corresponding bug report.

      Summary

      When emails contain text formatting, such as text coloring etc, the strip quote option does not appear to properly function anymore.

      Steps to Reproduce

      1. Create a Service Desk project
      2. Add a mail channel to the project
      3. Create an issue through mail
      4. Trigger a notification to be send from JIRA towards the customer account
      5. The notification will contain the —Write replies above—. Write a reply above this line, and apply some color changes to the text in the mail

      Expected Results

      The mail body gets added to the Service Desk request where only the text above the —Write replies above— gets added

      Actual Results

      The whole body content will get added to the request, including the original notification content originating from Service Desk.

      Notes

      In certain situations it appears duplicate comments are being created. The first comment added to the request displays only the content above the —Write replies above— line, whilst the "duplicate" comment adds the complete mail body.

        1. image-2016-12-12-11-09-36-845.png
          image-2016-12-12-11-09-36-845.png
          124 kB
        2. image-2016-12-15-21-21-53-376.png
          image-2016-12-15-21-21-53-376.png
          49 kB
        3. JSD email reply.txt
          17 kB
        4. mailheader.txt
          5 kB
        5. Outlook_Web_App.png
          Outlook_Web_App.png
          139 kB

            [JSDSERVER-4550] Strip quotes not working on incoming mails containing text formatting

            I have noticed in an earlier attachment that the following was present on the outputted wiki:

            —-—-—-— Reply above this line.
            

            The marker line is supposed to be on it's own line for the stripping to behave correctly.

            However, I can make a hardening approach that as long as the line starts with

            —-—-—-—
            

            Then it should be considered that strip the email below this. So with this logic, the above with the marker followed by some text, would be stripped.

            But if the line is say

            Some text —-—-—-—
            

            Then it should not be stripped, as while it can probably be assumed that it is meant to be stripped, stripping based on any text anywhere within a line, may have very unintended side effects.

            As well as this, I would encourage to vote for and watch https://jira.atlassian.com/browse/JSDSERVER-2356, as this feature would make it possible to add any custom rules to handle any specific cases that the default logic is not able to.

            Regards
            Matt

            Matthew McMahon (Inactive) added a comment - I have noticed in an earlier attachment that the following was present on the outputted wiki: —-—-—-— Reply above this line. The marker line is supposed to be on it's own line for the stripping to behave correctly. However, I can make a hardening approach that as long as the line starts with —-—-—-— Then it should be considered that strip the email below this. So with this logic, the above with the marker followed by some text, would be stripped. But if the line is say Some text —-—-—-— Then it should not be stripped, as while it can probably be assumed that it is meant to be stripped, stripping based on any text anywhere within a line, may have very unintended side effects. As well as this, I would encourage to vote for and watch https://jira.atlassian.com/browse/JSDSERVER-2356 , as this feature would make it possible to add any custom rules to handle any specific cases that the default logic is not able to. Regards Matt

            Apologies for the issues

            Stripping the text from an email is unfortunately quite a difficult process, especially when involving HTML, as to ensure the best results it has to rely on the following process:

            • Run through the text part of the email and strip that.
            • Then try to align the text with the html to work out where to strip the html part
            • When these don't line up, a fallback to running the strip rules over the converted wiki markup is used

            If anyone is having trouble still, can I please request that you raise a support request to provide some sample emails (raw source) of messages that were not stripped as expected.

            Some things to check also:

            • Email stripping only applies to comments. When a new ticket is created, no stripping is applied.
            • Email stripping is enabled on <instance>/secure/admin/SDMailInfo.jspa
            • The email in question, atleast includes the Service Desk (Reply above this line) marker, as that is the best fallback when involved with both text and wiki. The actual strip marker is the dashed line above this hint, that should be on its own new line.
            • Ensure that the project tickets by email are only being created via the Service Desk Mail Handler, and that there is no other JIRA Mail Handler also reading and processing emails, as this stripping logic is only used used by the Service Desk Mail Handler. If another handler is also creating issues/comments, then the stripping would not work as expected. One way to ensure this is to change the mailbox password and only update the Service Desk configuration.

            If these things have been checked, and the email was not stripped when should have expected it to, if I can get the source of the emails in question that were not stripped when added as a comment, will allow me to debug this particular edge case and harden the logic further.

            It might be possible to upload attachments on this ticket, but I would prefer to instead recommend raising a support request and linking back to this ticket. As this one is closed, would be better to have a new ticket created to address these particular issues.

            Regards
            Matt
            JIRA Service Desk developer

            Matthew McMahon (Inactive) added a comment - - edited Apologies for the issues Stripping the text from an email is unfortunately quite a difficult process, especially when involving HTML, as to ensure the best results it has to rely on the following process: Run through the text part of the email and strip that. Then try to align the text with the html to work out where to strip the html part When these don't line up, a fallback to running the strip rules over the converted wiki markup is used If anyone is having trouble still, can I please request that you raise a support request to provide some sample emails (raw source) of messages that were not stripped as expected. Some things to check also: Email stripping only applies to comments. When a new ticket is created, no stripping is applied. Email stripping is enabled on <instance>/secure/admin/SDMailInfo.jspa The email in question, atleast includes the Service Desk (Reply above this line) marker, as that is the best fallback when involved with both text and wiki. The actual strip marker is the dashed line above this hint, that should be on its own new line. Ensure that the project tickets by email are only being created via the Service Desk Mail Handler, and that there is no other JIRA Mail Handler also reading and processing emails, as this stripping logic is only used used by the Service Desk Mail Handler. If another handler is also creating issues/comments, then the stripping would not work as expected. One way to ensure this is to change the mailbox password and only update the Service Desk configuration. If these things have been checked, and the email was not stripped when should have expected it to, if I can get the source of the emails in question that were not stripped when added as a comment, will allow me to debug this particular edge case and harden the logic further. It might be possible to upload attachments on this ticket, but I would prefer to instead recommend raising a support request and linking back to this ticket. As this one is closed, would be better to have a new ticket created to address these particular issues. Regards Matt JIRA Service Desk developer

            We are running JSD 3.4.0 and still seeing this issue. It does not appear to be resolved even though this is marked as such. Very disappointing!!

            Jan Anderson added a comment - We are running JSD 3.4.0 and still seeing this issue. It does not appear to be resolved even though this is marked as such. Very disappointing!!

            Timothy Harris added a comment - - edited

            We are running JIRA 7.3.4 and SD 3.4.1 and this does not seem to work. The stripping of the previous parts of the thread do not work if response is in html. Works fine for plain text.

            Tested this with my gmail account.

            Timothy Harris added a comment - - edited We are running JIRA 7.3.4 and SD 3.4.1 and this does not seem to work. The stripping of the previous parts of the thread do not work if response is in html. Works fine for plain text. Tested this with my gmail account.

            Is this actually released? I see the same behavior?

            Timothy Harris added a comment - Is this actually released? I see the same behavior?

            Hi Matthew,

            This issue is crucial for our process, so could you please advise if there is an ETA for a fix on Cloud?

            Cheers.

            Ivan Tovbin added a comment - Hi Matthew, This issue is crucial for our process, so could you please advise if there is an ETA for a fix on Cloud? Cheers.

            A problem with the reply stripping has been identified, where a mail message has both text/plain and text/html parts, but the text content in both do not align.

            This is due to the stripping being performed on the text/plain part, and then aligned with the html content, to strip the text before the conversion to wiki markup occurs.

            A fix has been designed, to also run the stripper over the converted wiki markup, to pick up where this may occur and improve the result.

            In the meantime, the only workaround in situations where this is occurring, is to change the setting to not convert the HTML and instead only use the plain text part of incoming mail.

            Matthew McMahon (Inactive) added a comment - A problem with the reply stripping has been identified, where a mail message has both text/plain and text/html parts, but the text content in both do not align. This is due to the stripping being performed on the text/plain part, and then aligned with the html content, to strip the text before the conversion to wiki markup occurs. A fix has been designed, to also run the stripper over the converted wiki markup, to pick up where this may occur and improve the result. In the meantime, the only workaround in situations where this is occurring, is to change the setting to not convert the HTML and instead only use the plain text part of incoming mail.

            Kevin Liou added a comment - - edited

            Issue was tested and reproduced using Outlook Web App as a client. The environment was JIRA 7.2.3/JSD 3.2.4. Upgrading to JSD 3.2.6 did not change the behavior at all.

            The Strip Quotes functionality did not work at all while HTML Parsing was enabled, but worked once when it was disabled. Check out the below screenshot for an example.

            Kevin Liou added a comment - - edited Issue was tested and reproduced using Outlook Web App as a client. The environment was JIRA 7.2.3/JSD 3.2.4. Upgrading to JSD 3.2.6 did not change the behavior at all. The Strip Quotes functionality did not work at all while HTML Parsing was enabled, but worked once when it was disabled. Check out the below screenshot for an example.

            Tim H. added a comment -

            Matthew,

            I tried toggling strip quotes with HTML parsing enabled. The result appeared the same: no stripping seemed to occur in either case. When I turned off HTML parsing, the result looked much better:

            Tim H. added a comment - Matthew, I tried toggling strip quotes with HTML parsing enabled. The result appeared the same: no stripping seemed to occur in either case. When I turned off HTML parsing, the result looked much better:

            Hi thagen

            Thanks for the email.

            I have investigated it, and also tried to run through all the various tests, and flows. Whenever the JSD email stripping is enabled, the mail seems to be getting stripped where I would expect.

            This is quite an interesting case.

            All I can ask, is perhaps, toggle the email stripping back and forth, or attempt to turn off HTML stripping and see if the Text emails are stripped correctly. The text version of the mails has far less processing done to it, and therefore if the text emails are correctly stripped, then there must be a hidden error in the HTML logic.

            The other potential concern, is whether there are other (non JSD) mail handlers reading from the inbox and adding issues to the JSD project? The stripping library and configuration values are only relevant if the JSD incoming mail handlers are used. If there is another handler adding issues to the project, then this would do its own specific processing. If possible, it would be great to 100% rule out this possibility. The easiest way to achieve this in the past, if possible, has been to change the mailbox password and then only update it on the JSD incoming email channel.

            However if the problem persists, it would be recommended to raise a support request, to get some more personal assistance.

            Regards

            Matt

            Matthew McMahon (Inactive) added a comment - Hi thagen Thanks for the email. I have investigated it, and also tried to run through all the various tests, and flows. Whenever the JSD email stripping is enabled, the mail seems to be getting stripped where I would expect. This is quite an interesting case. All I can ask, is perhaps, toggle the email stripping back and forth, or attempt to turn off HTML stripping and see if the Text emails are stripped correctly. The text version of the mails has far less processing done to it, and therefore if the text emails are correctly stripped, then there must be a hidden error in the HTML logic. The other potential concern, is whether there are other (non JSD) mail handlers reading from the inbox and adding issues to the JSD project? The stripping library and configuration values are only relevant if the JSD incoming mail handlers are used. If there is another handler adding issues to the project, then this would do its own specific processing. If possible, it would be great to 100% rule out this possibility. The easiest way to achieve this in the past, if possible, has been to change the mailbox password and then only update it on the JSD incoming email channel. However if the problem persists, it would be recommended to raise a support request, to get some more personal assistance. Regards Matt

              Unassigned Unassigned
              mnassette MJ (Inactive)
              Affected customers:
              5 This affects my team
              Watchers:
              12 Start watching this issue

                Created:
                Updated:
                Resolved: