Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-83200

Cloud import is not working as expected because of the older AAID format used in Cloud

      Issue Summary

      During Cloud to Cloud migrations, sometimes importing the Cloud site to Data Center (DC) instance is required as a stepping stone for operations like merging two Cloud instances into one or making some adjustments in the user properties (emails, etc.) before moving to the destination Cloud instance.

      Cloud support two AAID formats as mentioned here: https://hello.atlassian.net/wiki/spaces/I/pages/1124693006/AAID+Format

      • 5a97df6a3724239813d0f607a
      • 342058:d6c19455-ae10-4a41-bad9-33d8d2d3c500

      The latter is causing issues in the DC as this is not a valid/expected format for DC to digest. Because of this, after importing the backup from the source Cloud into DC, users can't access contents that have relations with the userkey formatted with the latter format.

      This is reproducible on Data Center: yes

      Steps to Reproduce

      1. Create a Cloud site export containing user objects with the unexpected format and content related (created by, mentioned, etc.) to this type of user.
      2. Import it to the DC instance.
      3. Navigate to the content related to the user with unexpected userkey formatting

      Expected Results

      The content should be displayed without any issues.

      Actual Results

      The below exception is thrown in the atlassian-confluence.log file:

      2023-XX-XX XX:XX:XX,XXX ERROR [http-nio-8090-exec-4] [rest.api.model.ExceptionConverter] convertServiceException No status code found for exception, converting to internal server error : 
       -- url: /confluence/rest/experimental/search | userName: XXXXXX | referer: <confluence_baseURL>/index.action | traceId: 82fc7374bc3066e0
      java.lang.IllegalArgumentException: hexBinary needs to be even-length: 342058:d6c19455-ae10-4a41-bad9-33d8d2d3c500
           at javax.xml.bind.DatatypeConverterImpl.parseHexBinary(DatatypeConverterImpl.java:427)
          at javax.xml.bind.DatatypeConverter.parseHexBinary(DatatypeConverter.java:342)
          at com.atlassian.confluence.impl.search.v2.lucene.ContentPermissionSearchUtils.compressKey(ContentPermissionSearchUtils.java:138)
          at com.atlassian.confluence.impl.search.v2.lucene.ContentPermissionSearchUtils.getEncodedUserKey(ContentPermissionSearchUtils.java:126)
          at com.atlassian.confluence.impl.search.v2.lucene.filter.ContentPermissionsFilter.<init>(ContentPermissionsFilter.java:75)
      

      OR
      Enabling debug logs for the class com.atlassian.confluence.content.render.xhtml will show messages like:

      The ri:user tag contained an attribute that we could not process. The ri&#58;userkey attribute had a value of "557057&#58;07161c8c&#45;61af&#45;4a7c&#45;bc37&#45;7f46acca364e". This value could not be accepted for security reasons. We have chosen to remove this attribute from the tag and leave everything else in place so that we could process the input.
      

      Diagnostic

      To find out whether if your XML Backup from Confluence Cloud is impacted by this, you can:

      1. First find out if any user is exported with the impacted AAID Format in the extracted XML Backup:
        grep command for Unix Environment
        grep -A5 "<object class=\"ConfluenceUserImpl\" package=\"com.atlassian.confluence.user\">" entities.xml | grep key
        
        <id name="key"><![CDATA[5d668ffbc63b940e18c24375]]></id>
        <id name="key"><![CDATA[63fs0a7d6c30bbd7a33e35b1]]></id>
        <id name="key"><![CDATA[712120:879c7303-7ad0-4311-adb0-45573a2181ea]]></id>
        <id name="key"><![CDATA[712120:a7b4415d-f3d5-3cac-ac60-6eab1b5e291c]]></id>
        

        the last 2 AAID formats are impacted

      2. Then verify if there is any user-associated content to the impacted AAID Format in the extracted XML Backup:
        grep command for Unix Environment
        grep "ri:user ri:userkey" entities.xml
        
        <ri:user ri:userkey="5d668ffbc63b940e18c24375" />
        <ri:user ri:userkey="712120:a7b4415d-f3d5-3cac-ac60-6eab1b5e291c" />
        

        the second AAID format is impacted, and will be stripped away during XML Import

      Workaround

      Currently, there is no known workaround for this behavior. A workaround will be added here when available.

          Form Name

            [CONFSERVER-83200] Cloud import is not working as expected because of the older AAID format used in Cloud

            Adam Martin added a comment - - edited

            a2879c3b3278 - I'm still seeing this as problem when importing a Cloud "backup for Server", in to a fresh install of 8.5.17. I've replaced my actual cloud AAID with X's in the error message.

            	at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190)
            	at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
            	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63)
            	at java.base/java.lang.Thread.run(Thread.java:840)
            2024-11-25 20:00:50,423 ERROR [http-nio-8090-exec-4 url: /rest/experimental/search; user: <redacted username>] [rest.api.model.ExceptionConverter] convertServiceException No status code found for exception, converting to internal server error : 
             -- url: /rest/experimental/search | userName: <redacted username> | referer: http://localhost:8090/ | traceId: 9b35dd4a22a77a2b
            java.lang.IllegalArgumentException: hexBinary needs to be even-length: XXXXXX:XXXXXXXX-XXXX-XXXX-XXXX-XXX161747a0c
            	at javax.xml.bind.DatatypeConverterImpl.parseHexBinary(DatatypeConverterImpl.java:427)
            	at javax.xml.bind.DatatypeConverter.parseHexBinary(DatatypeConverter.java:342)
            	at com.atlassian.confluence.impl.search.v2.lucene.ContentPermissionSearchUtils.compressKey(ContentPermissionSearchUtils.java:138)
            	at com.atlassian.confluence.impl.search.v2.lucene.ContentPermissionSearchUtils.getEncodedUserKey(ContentPermissionSearchUtils.java:126) 

            This error is displayed when viewing the base URL and main activity feed.
            "Something's gone wrong.
            Try again later."

            Adam Martin added a comment - - edited a2879c3b3278 - I'm still seeing this as problem when importing a Cloud "backup for Server", in to a fresh install of 8.5.17. I've replaced my actual cloud AAID with X's in the error message. at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1190) at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:63) at java.base/java.lang. Thread .run( Thread .java:840) 2024-11-25 20:00:50,423 ERROR [http-nio-8090-exec-4 url: / rest /experimental/search; user: <redacted username>] [ rest .api.model.ExceptionConverter] convertServiceException No status code found for exception, converting to internal server error : -- url: / rest /experimental/search | userName: <redacted username> | referer: http: //localhost:8090/ | traceId: 9b35dd4a22a77a2b java.lang.IllegalArgumentException: hexBinary needs to be even-length: XXXXXX:XXXXXXXX-XXXX-XXXX-XXXX-XXX161747a0c at javax.xml.bind.DatatypeConverterImpl.parseHexBinary(DatatypeConverterImpl.java:427) at javax.xml.bind.DatatypeConverter.parseHexBinary(DatatypeConverter.java:342) at com.atlassian.confluence.impl.search.v2.lucene.ContentPermissionSearchUtils.compressKey(ContentPermissionSearchUtils.java:138) at com.atlassian.confluence.impl.search.v2.lucene.ContentPermissionSearchUtils.getEncodedUserKey(ContentPermissionSearchUtils.java:126) This error is displayed when viewing the base URL and main activity feed. "Something's gone wrong. Try again later."

            A fix for this issue is available in Confluence Server and Data Center 8.5.15.
            Upgrade now or check out the Release Notes to see what other issues are resolved.

            Jordan Anslow added a comment - A fix for this issue is available in Confluence Server and Data Center 8.5.15. Upgrade now or check out the Release Notes to see what other issues are resolved.

            A fix for this issue is available in Confluence Server and Data Center 9.0.3.
            Upgrade now or check out the Release Notes to see what other issues are resolved.

            Jordan Anslow added a comment - A fix for this issue is available in Confluence Server and Data Center 9.0.3. Upgrade now or check out the Release Notes to see what other issues are resolved.

            mb added a comment -

            Why is this only Priority Low? It has not been fixed since April 2023??

            This breaks our Confluence instance as we cannot mention users and cannot assign restrictions/permissions properly.

            See CSP-325818 & PSSRV-126117

            This makes us believe Atlassian does not have any interest in allowing the customers switch from Cloud to Datacenter if breaking bugs like this one don't get fixed or prioritized.

            mb added a comment - Why is this only Priority Low? It has not been fixed since April 2023?? This breaks our Confluence instance as we cannot mention users and cannot assign restrictions/permissions properly. See CSP-325818 & PSSRV-126117 This makes us believe Atlassian does not have any interest in allowing the customers switch from Cloud to Datacenter if breaking bugs like this one don't get fixed or prioritized.

            Syed Sajid added a comment - - edited

            Thanks  @Anton Pot. we followed  fix suggested in comment and issue resolved.

            Great work.

            Syed Sajid added a comment - - edited Thanks  @Anton Pot. we followed  fix suggested in comment and issue resolved. Great work.

            Ant Pot added a comment -

            We make a simple solution to fix this issue. Replace in entities.xml before import unique AAID with new_AAID supported by DC version.
            Save this code as python file, put you entities.xml to the same folder and run script. After process you have two new files in the folder. fix_entities.xml and unique_matching.json 
            Rename fix_entities.xml to entities.xml, put it in zip file (with replace) and start restore process.

            import sys
            import re
            import uuid
            import json
            import os
            # Set a default infile name
            default_work_file_name = 'entities.xml'
            # Regular expression pattern to match the desired text and capture both parts
            pattern = re.compile(r'(\d{3,6}):([a-f0-9]{8}-([a-f0-9]{4}-){3}[a-f0-9]{12})')
            # Check for command line arguments, use default if none provided
            work_file = sys.argv[1] if len(sys.argv) > 1 else default_work_file_name
            new_work_file = 'fix_' + work_file
            # Check if the infile exists
            if not os.path.exists(work_file):
                print(f"Error: The file '{work_file}' does not exist.")
                sys.exit(1)
            # Dictionary to store unique matches and their transformed texts
            unique_matches = {}
            # Process the file line by line
            with open(work_file, 'r', encoding='utf-8', errors='ignore') as file, open(new_work_file, 'w', encoding='utf-8') as outfile:
                for line in file:
                    for match in pattern.finditer(line):
                        original = f"{match.group(1)}:{match.group(2)}"
                        if original not in unique_matches:
                            # Replace the UUID part with uuid.hex[:24]
                            new_uuid_part = uuid.uuid4().hex[:24]
                            new_text = f"{new_uuid_part}"
                            unique_matches[original] = new_text        
            # Replace all occurrences of the original matches with new text in the current line
                    for original, new_text in unique_matches.items():
                        line = line.replace(original, new_text)        
            # Write the modified line to the new file
                    outfile.write(line)# Save the unique_matches dictionary as a JSON file
            with open('unique_matches.json', 'w') as json_file:
                json.dump(unique_matches, json_file, indent=4)
            # Now unique_matches contains all unique matches as keys and their transformed texts as values
            for original, new_text in unique_matches.items():
                print(f"Original: {original}")
                print(f"New: {new_text}") 

            {}P.S. I clearly not understand what we paying for? Atlassian didn't fix many bugs and still ignoring our issues.

            Ant Pot added a comment - We make a simple solution to fix this issue. Replace in entities.xml before import unique AAID with new_AAID supported by DC version. Save this code as python file, put you entities.xml to the same folder and run script. After process you have two new files in the folder. fix_entities.xml and unique_matching.json  Rename fix_entities.xml to entities.xml, put it in zip file (with replace) and start restore process. import sys import re import uuid import json import os # Set a default infile name default_work_file_name = 'entities.xml' # Regular expression pattern to match the desired text and capture both parts pattern = re.compile(r '(\d{3,6}):([a-f0-9]{8}-([a-f0-9]{4}-){3}[a-f0-9]{12})' ) # Check for command line arguments, use default if none provided work_file = sys.argv[1] if len(sys.argv) > 1 else default_work_file_name new_work_file = 'fix_' + work_file # Check if the infile exists if not os.path.exists(work_file):     print(f "Error: The file '{work_file}' does not exist." )     sys.exit(1) # Dictionary to store unique matches and their transformed texts unique_matches = {} # Process the file line by line with open(work_file, 'r' , encoding= 'utf-8' , errors= 'ignore' ) as file, open(new_work_file, 'w' , encoding= 'utf-8' ) as outfile:     for line in file:         for match in pattern.finditer(line):             original = f "{match.group(1)}:{match.group(2)}"             if original not in unique_matches:                 # Replace the UUID part with uuid.hex[:24]                 new_uuid_part = uuid.uuid4().hex[:24]                 new_text = f "{new_uuid_part}"                 unique_matches[original] = new_text        # Replace all occurrences of the original matches with new text in the current line         for original, new_text in unique_matches.items():             line = line.replace(original, new_text)        # Write the modified line to the new file         outfile.write(line)# Save the unique_matches dictionary as a JSON file with open( 'unique_matches.json' , 'w' ) as json_file:     json.dump(unique_matches, json_file, indent=4) # Now unique_matches contains all unique matches as keys and their transformed texts as values for original, new_text in unique_matches.items():     print(f "Original: {original}" )     print(f "New: {new_text}" ) { }P.S. I clearly not understand what we paying for? Atlassian didn't fix many bugs and still ignoring our issues.

              f8dbc9c9ac45 Anshul Chokhani
              9f7de485df51 Basar Beykoz
              Affected customers:
              24 This affects my team
              Watchers:
              36 Start watching this issue

                Created:
                Updated:
                Resolved: