Uploaded image for project: 'Jira Service Management Data Center'
  1. Jira Service Management Data Center
  2. JSDSERVER-5096

Invalid characters encountered in base64 data

XMLWordPrintable

      Issue Summary

      Service Desk email channels may get stuck and fail to process any more emails and looking at the email processor and scheduler, you would see that JIRA is still attempting to pull and process emails however no new email are being processed.

      Looking at the processing log, it would appear that JIRA has stopped processing emails however when you look at the log proper in particular after setting the incoming mail to Debug mode, it would be observed that the JIRA Service Desk mail channels are still trying to pull emails but they MailJobRunner fails because of:

      Invalid charactersencountered in base64 data.

      2017-05-05 03:45:48,228 Caesium-1-3 ERROR ServiceRunner     [c.a.m.incoming.jepp.processor] MailJobRunner Failed
java.lang.RuntimeException: Unable to read next message
	at com.atlassian.jira.internal.mail.processor.feature.puller.mailitem.streaming.MailItemDTOIterator.next(MailItemDTOIterator.java:51)
	at com.atlassian.jira.internal.mail.processor.feature.puller.mailitem.streaming.MailItemDTOIterator.next(MailItemDTOIterator.java:15)
	at com.atlassian.jira.internal.mail.processor.feature.processor.MailProcessorWorker.process(MailProcessorWorker.java:75)
	at com.atlassian.jira.internal.mail.processor.feature.processor.MailProcessorService.run(MailProcessorService.java:36)
	at com.atlassian.jira.internal.mail.processor.services.MailProcessorExecutor.run(MailProcessorExecutor.java:30)
	at com.atlassian.jira.internal.mail.processor.services.AbstractMailExecutor.execute(AbstractMailExecutor.java:42)
	at com.atlassian.jira.internal.mail.processor.services.MailJobRunner.runJob(MailJobRunner.java:35)
	at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:153)
	at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:118)
	at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:97)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:401)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:396)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:349)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:255)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:252)
	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:65)
	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:59)
	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:34)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: invalid characters encountered in base64 data
	at org.bouncycastle.util.encoders.Base64Encoder.decode(Unknown Source)
	at com.atlassian.jira.internal.mail.processor.feature.puller.mailitem.streaming.MailItemTupleConsumer.decodeChunk(MailItemTupleConsumer.java:82)
	at com.atlassian.jira.internal.mail.processor.feature.puller.mailitem.streaming.MailItemTupleConsumer.consumeRow(MailItemTupleConsumer.java:65)
	at com.atlassian.jira.internal.mail.processor.feature.puller.mailitem.streaming.MailItemDTOIterator.readNextMessageFully(MailItemDTOIterator.java:71)
	at com.atlassian.jira.internal.mail.processor.feature.puller.mailitem.streaming.MailItemDTOIterator.next(MailItemDTOIterator.java:47)
      Environment

      This error from all occurrences seems to affect Microsoft Exchange Server.

      Diagnosis
      1. When you view the processing log tab, there would be no sign of JIRA attempting to process any new emails.
      2. When you view the log, you would see the error above.
      Expected Behavior

      JIRA would be able to ignore this email and continue to process other mails rather than having this one email blocking the whole mail channel.

      Actual Behavior

      JIRA would not be processing any new emails and keep throwing the error mentioned.

      Workaround

      The problem causing message must be removed in order for the mail processor to process any further mail.

      To remove the message:

      1. Identify the ID of the message (mail item) that needs to be removed (see kb article)
      2. Remove the mail item from the database: 
        • delete from "AO_2C4E5C_MAILITEMCHUNK" where "MAIL_ITEM_ID" = <id to delete>;
delete from "AO_2C4E5C_MAILITEM" where "ID" = <id to delete>;
        • These commands must be performed in this order.
      Alternate Workaround

      Delete the JIRA email channel and create it again.

      This however would still leave the old emails in the database but JIRA would be processing the emails from the new email channel id.

            [JSDSERVER-5096] Invalid characters encountered in base64 data

            Dastin Kuwałek [SoftwarePlant] added a comment -

            Looking forward to the fix as well =) it has happened few times last years so I will  feel safer after I upgrade to a version with the fix! 

            Dastin Kuwałek [SoftwarePlant] added a comment - Looking forward to the fix as well =) it has happened few times last years so I will  feel safer after I upgrade to a version with the fix! 

            Anton Kaiser added a comment -

            Also affects 3.12.2. Looking forward to the new release with the fix!

            Anton Kaiser added a comment - Also affects 3.12.2. Looking forward to the new release with the fix!

            Rian Spivak added a comment -

            We have hit this bug on our older version of Service Desk and the biggest pain is the lack of errors in the interface about this happening. The only place to find this is in the logs on the server. My current workaround fix to process emails again is to look at the "AO_2C4E5C_MAILITEM" table and find the oldest item with the "NEW" status. This is the one holding up the processing. I then record the ID of the email. Then I update the status to "PROCESSED". Something like:

            UPDATE AO_2C4E5C_MAILITEM set status='PROCESSED' where id='<ID OF MAILITEM>'

            This should bypass that email and continue processing any further queued emails. To identify which email was causing the issue retrieve the mail chunk value from the AO_2C4E5C_MAILITEMCHUNK table using the id recorded for the mail_item_id column.

            select * from AO_2C4E5C_MAILITEMCHUNK where mail_item_id='<ID OF MAILITEM>'

            Copy the value for the mime_msg_chunk and run it through a base64 decode to view the email.

             

            Rian Spivak added a comment - We have hit this bug on our older version of Service Desk and the biggest pain is the lack of errors in the interface about this happening. The only place to find this is in the logs on the server. My current workaround fix to process emails again is to look at the "AO_2C4E5C_MAILITEM" table and find the oldest item with the "NEW" status. This is the one holding up the processing. I then record the ID of the email. Then I update the status to "PROCESSED". Something like: UPDATE AO_2C4E5C_MAILITEM set status= 'PROCESSED' where id= '<ID OF MAILITEM>' This should bypass that email and continue processing any further queued emails. To identify which email was causing the issue retrieve the mail chunk value from the AO_2C4E5C_MAILITEMCHUNK table using the id recorded for the mail_item_id column. select * from AO_2C4E5C_MAILITEMCHUNK where mail_item_id= '<ID OF MAILITEM>' Copy the value for the mime_msg_chunk and run it through a base64 decode to view the email.  

            David Luke added a comment -

            affects version 3.9.4, as well, by the way.

            David Luke added a comment - affects version 3.9.4, as well, by the way.

            Tim Evans (Inactive) added a comment -

            dave19 an ETA is not currently available, please continue to watch this bug for updates. A fix may be part of a new version of JSD and if that is the case then yes you will need to upgrade JIRA itself (by running the installer) so that the version of JIRA will be compatible with the version of JSD containing the fix.

            lance.milleson1 the "offending email" would be the one that JSD tries to read and process over and over again. It is likely the oldest unprocessed message (status = New) when looking at the "AO_2C4E5C_MAILITEM" database table. 

            • Your expectations make sense. You are welcome to create a support case so our team can look into why you didn't see what you expected and why this impacts 4 of your 5 service desks.

            Tim Evans (Inactive) added a comment - dave19 an ETA is not currently available, please continue to watch this bug for updates. A fix may be part of a new version of JSD and if that is the case then yes you will need to upgrade JIRA itself (by running the installer) so that the version of JIRA will be compatible with the version of JSD containing the fix. lance.milleson1 the "offending email" would be the one that JSD tries to read and process over and over again. It is likely the oldest unprocessed message (status = New) when looking at the "AO_2C4E5C_MAILITEM" database table.  Your expectations make sense. You are welcome to create a support case so our team can look into why you didn't see what you expected and why this impacts 4 of your 5 service desks.

            Lance Milleson added a comment - - edited

            How do you find the offending e-mail?

            I saved all the content from the table into individual base64 files, then used the Windows certutil to convert them to emls. I then re-sent them to Service Desk and the problem didn't happen.

            I expected:

            1) converting them from base64 to eml would have given an error or

            2) sending them to service desk would have caused the issue, again

             

            Also strange is that we have 5 Service Desks and, when the problem happens (now twice), one of the desks is not affected.  But the other 4 stop processing email requests.

            Lance Milleson added a comment - - edited How do you find the offending e-mail? I saved all the content from the table into individual base64 files, then used the Windows certutil to convert them to emls. I then re-sent them to Service Desk and the problem didn't happen. I expected: 1) converting them from base64 to eml would have given an error or 2) sending them to service desk would have caused the issue, again   Also strange is that we have 5 Service Desks and, when the problem happens (now twice), one of the desks is not affected.  But the other 4 stop processing email requests.

            David Luke added a comment -

            Glad to see this is now In Progress. Can any more light be shed (like expected resolved date? and will I have to upgrade Jira itself to avail myself of the fix?)? This is an Urgent priority issue for me.

            David Luke added a comment - Glad to see this is now In Progress. Can any more light be shed (like expected resolved date? and will I have to upgrade Jira itself to avail myself of the fix?)? This is an Urgent priority issue for me.

            Christian Bär added a comment - - edited

            "Don't f... the customer."

            What else is it, if there is an obvious Bug causing data loss and nobody cares?

            I can't tell how pissed I'm of this behavior - is Atlasians goal to become the next Oracle or Microsoft of the 90s ?

            What should I tell my customer I'm serving as a platinum partner? Oh sorry, there are too few people running in this problem, so that almighty Atlassian is not willing to deal with it?

            It may have to do with microsoft exchange but really, if there is a corrupted email - have you ever heard the term fault tolerance?

            And also, I can argue for Atlassian regarding any feature request, Atlassian is not even reacting to. No problem at all. That's agile.

            But this is a BUG. If there is one jsd project receiving that corrupted email, ALL jsd projects fail to receive emails. AND after droping and recreating the configuration, there are Emails missing. People thinking their service request is beeing processed and waiting are - think what - pissed. "What shit of a piece of software is that?" they ask. Ah , uh, it's Atlassian.

            So, you are pissed of my rant. Good. But now think over again. I am right.

            Christian Bär added a comment - - edited "Don't f... the customer." What else is it, if there is an obvious Bug causing data loss and nobody cares? I can't tell how pissed I'm of this behavior - is Atlasians goal to become the next Oracle or Microsoft of the 90s ? What should I tell my customer I'm serving as a platinum partner? Oh sorry, there are too few people running in this problem, so that almighty Atlassian is not willing to deal with it? It may have to do with microsoft exchange but really, if there is a corrupted email - have you ever heard the term fault tolerance? And also, I can argue for Atlassian regarding any feature request, Atlassian is not even reacting to. No problem at all. That's agile. But this is a BUG. If there is one jsd project receiving that corrupted email, ALL jsd projects fail to receive emails. AND after droping and recreating the configuration, there are Emails missing. People thinking their service request is beeing processed and waiting are - think what - pissed. "What shit of a piece of software is that?" they ask. Ah , uh, it's Atlassian. So, you are pissed of my rant. Good. But now think over again. I am right.

            Luigi Ferri added a comment -

            Also the 3.6.1 is impacted. Debugging the issue my colleagues told me that just one e-mail with "Invalid characters base64 data"  can stop the incoming mails. For a consumer service desk this is a real issue. Please kindly fix it.

            Luigi Ferri added a comment - Also the 3.6.1 is impacted. Debugging the issue my colleagues told me that just one e-mail with " Invalid characters base64 data"   can stop the incoming mails. For a consumer service desk this is a real issue. Please kindly fix it.

            Mats Erksén added a comment -

            Also version 3.8.3 is affected by this

            Mats Erksén added a comment - Also version 3.8.3 is affected by this

              cpollock Christopher Pollock (Inactive)
              ijimoh Ismael Olusula Jimoh (Inactive)
              Affected customers:
              26 This affects my team
              Watchers:
              36 Start watching this issue

                Created:
                Updated:
                Resolved: