Importing data in to Postgres with NUL characters fails

XMLWordPrintable

    • 6
    • 4
    • Severity 3 - Minor
    • 0

      NOTE: This bug report is for JIRA Server. Using JIRA Cloud? See the corresponding bug report.

      A customer recently attempted an import of their BTF data in to OnDemand.

      Their environment was as follows:

           JIRA Build                                    : 6.0#6095-sha1:601557e8c67ba1d610a3be6ada7f17edcd3dfd76
     Build Date                                    : Sun May 19 00:00:00 EST 2013
     JIRA Installation Type                        : Standalone
     Java Version                                  : 1.7.0_13 - Oracle Corporation
// snip memory params
     JVM Input Arguments                           : -Djava.util.logging.config.file=/usr/local/jira/current/conf/logging.properties -XX:MaxPermSize=256m -Xms256m -Xmx768m -Djava.awt.headless=true -Datlassian.standalone=JIRA -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -Dmail.mime.decodeparameters=true -XX:+PrintGCDateStamps -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/usr/local/jira/current/endorsed -Dcatalina.base=/usr/local/jira/current -Dcatalina.home=/usr/local/jira/current -Djava.io.tmpdir=/usr/local/jira/current/temp
     Java Compatibility Information                : JIRA version = 6.0, Java Version = 1.7.0_13

___ Database Configuration _________________

     Loading entityengine.xml from                 : file:/usr/local/jira/current/atlassian-jira/WEB-INF/classes/entityengine.xml
     Entity model field type name                  : mysql
     Entity model schema name                      : 
     Database Version                              : MySQL - 5.1.49-3
     Database Driver                               : MySQL-AB JDBC Driver - mysql-connector-java-5.1.10 ( Revision: ${svn.Revision} )
     Database URL                                  : jdbc:mysql://localhost/jiradb?useUnicode=true&characterEncoding=utf8&sessionVariables=storage_engine=InnoDB
     Database JDBC config                          : mysql jdbc:mysql://localhost/jiradb?useUnicode=true&characterEncoding=utf8&sessionVariables=storage_engine=InnoDB

      And somewhere in one of their issue's comments, they had some NUL characters:

      $ egrep -n '\x00' entities.xml | cat -v
203257:		 0. req self.addDevice(label: ^@^@ (TwoCharId), config:  0x0000: 00016f39                            | ..o9^M

      When importing this data in to JIRA + Postgres, a GenericEntityException is thrown, caused by a SQL exception: 

      SQL Exception while executing the following:INSERT INTO public.jiraaction (ID, issueid, AUTHOR, actiontype, actionlevel, rolelevel, actionbody, CREATED, UPDATEAUTHOR, UPDATED, actionnum) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) (ERROR: invalid byte sequence for encoding "UTF8": 0x00)

      I'm not sure what we can do to make JIRA more resilient when encountering NUL characters, or what the "correct" thing to do here is (e.g., keep the NUL characters, discard, rewrite, ...). Alls I know is that it's a difficult error to triage and diagnose for us, let alone our customers.

      Workaround

      Running the following command over the entities.xml file from the backup and re-archiving it results in being able to import the data in JIRA: 

      perl -pi -e 's/\x00//g' entities.xml

              Assignee:
              Unassigned
              Reporter:
              Oswaldo Hernandez (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: