Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-18509

MySQL databases incapable of handling 4byte UTF-8 Characters. Confluence should handle this gracefully

      NOTE: This suggestion is for Confluence Server. Using Confluence Cloud? See the corresponding suggestion.

      Atlassian Update - April 2020

      The Confluence server development team has now completed both item 1 (Support for emoji in MySQL) and item 2 (providing meaningful error messages). We strongly encourage customers to follow the steps outlined in our documentation to migrate from UTF8 to UTF8MB4, completely resolving this issue.

      If you are unable to migrate to UTF8MB4 we have added a number of meaningful error messages to Confluence to stop users attempting to add unsupported 4-byte characters. We have added these errors to:

      • page publish/save events for pages and blogs 
      • page titles
      • comment/inline comment saves.

      Please note that we have not added this error message to content fields that are less used/less likely to contain 4byte characters including:

      • comments accessed via email,
      • the editor in Questions for Confluence,
      • comments on attachments,
      • gadget comments,
      • attachment titles
      • content added via Confluence Mobile. 

      From the Confluence development team, we want to say thank you for everyone who provided feedback as part of both of the fixes mentioned above. We hope you now enjoy an emoji-filled Confluence. 

      Regards,

      The Confluence Server Team

      The following error:

      org.springframework.jdbc.UncategorizedSQLException: Hibernate operation: could not update: com.atlassian.confluence.core.BodyContent#38141954; uncategorized SQLException for SQL []; SQL state [HY000]; error code [1366]; Incorrect string value: '\xF4\x80\x82\x84 z...' for column 'BODY' at row 1; nested exception is java.sql.SQLException: Incorrect string value: '\xF4\x80\x82\x84 z...' for column 'BODY' at row 1
      Caused by: java.sql.SQLException: Incorrect string value: '\xF4\x80\x82\x84 z...' for column 'BODY' at row 1
      at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1055)
      at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:956)
      at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3536)
      

      is thrown by confluence when your try to save a page with bullet points copied from an external source (Pdf,etc).

      It appears to be directly related to this MySQL limitation: http://bugs.mysql.com/bug.php?id=28853

      We should handle this gracefully since it is a known limitation for MySQL.

      Note: This errors can also happen in the following situations:

      1. Restoring an XML backup into a MySQL database
      2. Attempting to log into Confluence
        For the login failure, the error message will look like this:
        url:/dologin.action] [engine.jdbc.spi.SqlExceptionHelper] logExceptions Incorrect string value: '\xF0\x9F\x98\x8E p...' for column 'description' at row 1
        
      Workaround for XML imports:

      Use XML cleaner from this KB.

      Also documented here

      Workaround for login issues

      If you are seeing the Incorrect string value errors for the /dologin.action requests, proceed as follows:

      1. Find the offending groups on the AD side and remove the 4-byte characters from the description based on the memberships of the user that cannot log in, or
      2. Change the Description attribute to the CN instead on the directory settings in Confluence

            [CONFSERVER-18509] MySQL databases incapable of handling 4byte UTF-8 Characters. Confluence should handle this gracefully

            Hi ff7cbaf6d4e6

            Are you talking about 6.13.5? if yes, then it doesn't support utf8mb4 and won't do it in future.
            However, meaningful error messages will be provided soon on 6.13.x by way of backporting fixes.

            Also, please see the following item for utf8mb4 support: https://jira.atlassian.com/browse/CONFSERVER-53642

            Thanks,
            Ganesh

            Ganesh Gautam added a comment - Hi ff7cbaf6d4e6 Are you talking about 6.13.5? if yes, then it doesn't support utf8mb4 and won't do it in future. However, meaningful error messages will be provided soon on 6.13.x by way of backporting fixes. Also, please see the following item for utf8mb4 support:  https://jira.atlassian.com/browse/CONFSERVER-53642 Thanks, Ganesh

            5140883 added a comment -

            Hi philippe.perez, we got the same error with 6.18.5, can I update mysql setting to UTF8MB4?

            In doc, it says only confluence with 7.2 or later version can upgrade. Confuse...

            5140883 added a comment - Hi  philippe.perez , we got the same error with 6.18.5, can I update mysql setting to UTF8MB4? In doc, it says only confluence with 7.2 or later version can upgrade. Confuse...

            Hi philippe.perez, We are looking into backporting the meaningful error messages to 6.13. We will update this ticket once that is completed. Unfortunately, if you are interested in full emoji support in MySQL you will need to upgrade to 7.3 or our upcoming Enterprise Release 7.4. 

            Matthew Saxby (Inactive) added a comment - Hi philippe.perez , We are looking into backporting the meaningful error messages to 6.13. We will update this ticket once that is completed. Unfortunately, if you are interested in full emoji support in MySQL you will need to upgrade to 7.3 or our upcoming Enterprise Release 7.4. 

            Hello,

            When will the step 2 be provided in the current LTS version ? That is the 6.13, which is still supported by the Enterprise license so candidate for such backport.

            Thanks.

             

            Philippe PEREZ added a comment - Hello, When will the step 2 be provided in the current LTS version ? That is the 6.13, which is still supported by the Enterprise license so candidate for such backport. Thanks.  

            Hi All,

            I wanted to follow up on our update above. The Confluence server development team has now completed both item 1 (Support for emoji in MySQL) and item 2 (providing meaningful error messages). We strongly encourage customers to follow the steps outlined in our documentation to migrate from UTF8 to UTF8MB4, completely resolving this issue.

            If you are unable to migrate to UTF8MB4 we have added a number of meaningful error messages to Confluence to stop users attempting to add unsupported 4-byte characters. We have added these errors to:

            • page publish/save events for pages and blogs 
            • page titles
            • comment/inline comment saves.

            Please note that we have not added this error message to content fields that are less used/less likely to contain 4byte characters including:

            • comments accessed via email,
            • the editor in Questions for Confluence,
            • comments on attachments,
            • gadget comments,
            • attachment titles
            • content added via Confluence Mobile. 

            From the Confluence development team, we want to say thank you for everyone who provided feedback as part of both of the fixes mentioned above. We hope you now enjoy an emoji-filled Confluence. 

            Regards,

            The Confluence Server Team

            Matthew Saxby (Inactive) added a comment - Hi All, I wanted to follow up on our update above. The Confluence server development team has now completed both item 1 (Support for emoji in MySQL) and item 2 (providing meaningful error messages). We strongly encourage customers to follow the  steps outlined in our documentation  to migrate from UTF8 to UTF8MB4, completely resolving this issue. If you are unable to migrate to UTF8MB4 we have added a number of meaningful error messages to Confluence to stop users attempting to add unsupported 4-byte characters. We have added these errors to: page publish/save events for pages and blogs  page titles comment/inline comment saves. Please note that we have not added this error message to content fields that are less used/less likely to contain 4byte characters including: comments accessed via email, the editor in Questions for Confluence, comments on attachments, gadget comments, attachment titles content added via Confluence Mobile.  From the Confluence development team, we want to say thank you for everyone who provided feedback as part of both of the fixes mentioned above. We hope you now enjoy an emoji-filled Confluence.  Regards, The Confluence Server Team

            Ganesh Gautam added a comment - - edited

            Atlassian Update - Feb 2020
            We have been working on a fix for this issue and wanted to give you an update on where we are up to. Our work in broken down into two key milestones:

            1) Support for emoji in MySQL: We have delivered support for UTF8MB4 in Confluence 7.3.1. This work was being tracked in CONFSERVER-53642 - Support MySQL Databases using utf8mb4 and is now Done. Confluence now supports 4 byte unicode characters with MySQL UTF8MB4 collation. To assist customers with existing MySQL databases (that do not support UTF8MB4) we have provided a detailed guide outlining instructions on how to move to a UTF8MB4 compatible MySQL database here.

            2) Provide meaningful error messages when adding emoji on instances still with MySQL utf8: To stop users attempting to add unsupported Emoji to Confluence with MySQL utf8, we have added verification to publish/save events for all pages and blogs. This will provide users with a useful message instructing them to remove the character in order to save their content. We expect the message to look something like this:


            We have released this change as part of 7.3.1 and will backport this change to the next bugfix release of 6.13 as well. Note we also plan to add this error message to emoji’s included in page titles, comments, inline comments and attachment comments. These will be added in future bugfix releases as they become available. You can continue to track this work in this Jira ticket.

            Cheers,
            The Confluence Server Team

            Ganesh Gautam added a comment - - edited Atlassian Update - Feb 2020 We have been working on a fix for this issue and wanted to give you an update on where we are up to. Our work in broken down into two key milestones: 1) Support for emoji in MySQL: We have delivered support for UTF8MB4 in Confluence 7.3.1. This work was being tracked in CONFSERVER-53642  - Support MySQL Databases using utf8mb4 and is now Done. Confluence now supports 4 byte unicode characters with MySQL UTF8MB4 collation. To assist customers with existing MySQL databases (that do not support UTF8MB4) we have provided a detailed guide outlining instructions on how to move to a UTF8MB4 compatible MySQL database here . 2) Provide meaningful error messages when adding emoji on instances still with MySQL utf8: To stop users attempting to add unsupported Emoji to Confluence with MySQL utf8, we have added verification to publish/save events for all pages and blogs. This will provide users with a useful message instructing them to remove the character in order to save their content. We expect the message to look something like this: We have released this change as part of 7.3.1 and will backport this change to the next bugfix release of 6.13 as well. Note we also plan to add this error message to emoji’s included in page titles, comments, inline comments and attachment comments. These will be added in future bugfix releases as they become available. You can continue to track this work in this Jira ticket. Cheers, The Confluence Server Team

            Works for me as well.  Here's how to update your database.  Stop confluence then update my.cnf:

             

            [client]
            default-character-set = utf8mb4
            [mysqld]
            character-set-server = utf8mb4
            

             

            Reload or restart mysqld.  Next update all of your tables (there are many):

             

            mysql> SET FOREIGN_KEY_CHECKS = 0;
            mysql> ALTER TABLE <all of your tables> convert TO CHARACTER SET utf8mb4 COLLATE utf8mb4_bin;
            mysql> SET FOREIGN_KEY_CHECKS = 1;

             

            Start confluence.  Process is similar for Jira - change your database driver from mysql to mysql57 first.  Enjoy 💩

             

            Nik Ambrosch added a comment - Works for me as well.  Here's how to update your database.  Stop confluence then update my.cnf:   [client] default-character-set = utf8mb4 [mysqld] character-set-server = utf8mb4   Reload or restart mysqld.  Next update all of your tables (there are many):   mysql> SET FOREIGN_KEY_CHECKS = 0; mysql> ALTER TABLE <all of your tables> convert TO CHARACTER SET utf8mb4 COLLATE utf8mb4_bin; mysql> SET FOREIGN_KEY_CHECKS = 1;   Start confluence.  Process is similar for Jira - change your database driver from mysql to mysql57 first.  Enjoy 💩  

            I upgraded to Confluence 7.3.1 and had already used MySQL 5.7.29. 

            Christian König added a comment - I upgraded to Confluence 7.3.1 and had already used MySQL 5.7.29. 

            Good to know.

            That means that you also upgraded to Confluence 7.3.1 and your database was in 5.9 ?

            Philippe PEREZ added a comment - Good to know. That means that you also upgraded to Confluence 7.3.1 and your database was in 5.9 ?

            Did an update yesterday and it worked fine. Don't forget to update database, tables and columns.

            https://confluence.atlassian.com/confkb/how-to-fix-the-collation-and-character-set-of-a-mysql-database-670958160.html

             

            Christian König added a comment - Did an update yesterday and it worked fine. Don't forget to update database, tables and columns. https://confluence.atlassian.com/confkb/how-to-fix-the-collation-and-character-set-of-a-mysql-database-670958160.html  

              ggautam Ganesh Gautam
              pkamal Partha
              Affected customers:
              185 This affects my team
              Watchers:
              176 Start watching this issue

                Created:
                Updated:
                Resolved: