Uploaded image for project: 'Crucible'
  1. Crucible
  2. CRUC-681

Be able to export the projects on one server and import them into another

    • 310
    • 2
    • Our product teams collect and evaluate feedback from a number of different sources. To learn more about how we use customer feedback in the planning process, check out our new feature policy.

      The ability to move a project from one of their crucible servers to another (which houses its own crucible projects).

      A function similar to confluence export/import space would allow customers to easily do this

            [CRUC-681] Be able to export the projects on one server and import them into another

            Using the Project transfer for Crucible app, a project is imported with the review, but the author's comments & date are replaced with wrong information (admin account and date/hour of the import), as shown in the screenshot below:

            Steps to reproduce

            1. Create a new Crucible project.
            2. With a user with admin rights (e.g., admin):
              • Create a review on that project
              • Add some comments
              • Add a normal user (e.g., test) as a reviewer.
            3. With the test user account, add some comments to the review
            4. Export the repository with the user with admin rights.
            5. Import the project With the same admin user in a replica instance (same user base, permission scheme, etc.).

            Andrew Bimba added a comment - Using the Project transfer for Crucible app , a project is imported with the review, but the author's comments & date are replaced with wrong information (admin account and date/hour of the import), as shown in the screenshot below: Steps to reproduce Create a new Crucible project. With a user with admin rights (e.g., admin): Create a review on that project Add some comments Add a normal user (e.g., test) as a reviewer. With the test user account, add some comments to the review Export the repository with the user with admin rights. Import the project With the same admin user in a replica instance (same user base, permission scheme, etc.).

            Running into an issue where the migration plugin cannot create a review on the target instance, but it does not pump enough information out in the Debug log to really see what the issue is.

            It's creating it from the migration structure:

            ./PEC-1422/review.xml

            ./PEC-1422/items/CFR-4744.xml

            ./PEC-1422/items/CFR-4745.xml

            ./PEC-1422/items/CFR-4746.xml

            ./PEC-1422/items/CFR-4747.xml

            ./PEC-1422/items/CFR-4748.xml

            ./PEC-1422/items/CFR-4749.xml

            ./PEC-1422/items/CFR-4750.xml

            ./PEC-1422/items/CFR-4751.xml

            It actually does create the review and populates it with initial data (review participants, summary, description, linked Jira issue keys, etc.) but then fails to import the CFRs, the stacktrace just says:

            2022-07-12 19:12:42,184 DEBUG [qtp2085745483-6582 ] com.atlassian.crucible.plugins.transfer.impl.ImportProjectJob ImportProjectJob-importReviewData - Created review 'PEC-1422' as 'PEC-441'

            ...

            2022-07-12 19:12:42,348 DEBUG [qtp2085745483-6582 ] com.atlassian.crucible.plugins.transfer.impl.ImportProjectJob ImportProjectJob-addReviewItems - Importing review item data from '/opt/rtn/atlassian/fecru/var/tmp/crucible-import8392049506580723607/PEC-1422'

            2022-07-12 19:12:42,417 DEBUG [qtp2085745483-6582 ] fisheye TxHelperStylePlatformTransactionManager-rollback - Rolled back transaction started from:

            java.lang.Throwable: (Stack trace at transaction depth 0 start)

            ....

            2022-07-12 19:12:42,538 ERROR [qtp2085745483-6582 ] com.atlassian.crucible.plugins.transfer.impl.ImportProjectJob ImportProjectJob-run - Failed to import project 'PEC'
            java.lang.IllegalArgumentException: the given review item is not valid.
                    at com.atlassian.crucible.spi.impl.DefaultReviewService.lambda$addReviewItem$13(DefaultReviewService.java:886) [fisheye.jar:?]
                    at com.atlassian.fisheye.spi.impl.DefaultTxTemplate.execute(DefaultTxTemplate.java:78) [fisheye.jar:?]
                    at com.atlassian.crucible.spi.impl.DefaultReviewService.addReviewItem(DefaultReviewService.java:864) [fisheye.jar:?]
                    at sun.reflect.GeneratedMethodAccessor3822.invoke(Unknown Source) [?:?]
                    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [?:1.8.0_332]
                    at java.lang.reflect.Method.invoke(Method.java:498) [?:1.8.0_332] 

             

            Any idea what 'java.lang.IllegalArgumentException: the given review item is not valid.' is really being caused by in this case?

            Asset Management added a comment - Running into an issue where the migration plugin cannot create a review on the target instance, but it does not pump enough information out in the Debug log to really see what the issue is. It's creating it from the migration structure: ./PEC-1422/review.xml ./PEC-1422/items/CFR-4744.xml ./PEC-1422/items/CFR-4745.xml ./PEC-1422/items/CFR-4746.xml ./PEC-1422/items/CFR-4747.xml ./PEC-1422/items/CFR-4748.xml ./PEC-1422/items/CFR-4749.xml ./PEC-1422/items/CFR-4750.xml ./PEC-1422/items/CFR-4751.xml It actually does create the review and populates it with initial data (review participants, summary, description, linked Jira issue keys, etc.) but then fails to import the CFRs, the stacktrace just says: 2022-07-12 19:12:42,184 DEBUG [qtp2085745483-6582 ] com.atlassian.crucible.plugins.transfer.impl.ImportProjectJob ImportProjectJob-importReviewData - Created review 'PEC-1422' as 'PEC-441' ... 2022-07-12 19:12:42,348 DEBUG [qtp2085745483-6582 ] com.atlassian.crucible.plugins.transfer.impl.ImportProjectJob ImportProjectJob-addReviewItems - Importing review item data from '/opt/rtn/atlassian/fecru/var/tmp/crucible-import8392049506580723607/PEC-1422' 2022-07-12 19:12:42,417 DEBUG [qtp2085745483-6582 ] fisheye TxHelperStylePlatformTransactionManager-rollback - Rolled back transaction started from: java.lang.Throwable: (Stack trace at transaction depth 0 start) .... 2022-07-12 19:12:42,538 ERROR [qtp2085745483-6582 ] com.atlassian.crucible.plugins.transfer.impl.ImportProjectJob ImportProjectJob-run - Failed to import project 'PEC' java.lang.IllegalArgumentException: the given review item is not valid.         at com.atlassian.crucible.spi.impl.DefaultReviewService.lambda$addReviewItem$13(DefaultReviewService.java:886) [fisheye.jar:?]         at com.atlassian.fisheye.spi.impl.DefaultTxTemplate.execute(DefaultTxTemplate.java:78) [fisheye.jar:?]         at com.atlassian.crucible.spi.impl.DefaultReviewService.addReviewItem(DefaultReviewService.java:864) [fisheye.jar:?]         at sun.reflect.GeneratedMethodAccessor3822.invoke(Unknown Source) [?:?]         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [?:1.8.0_332]         at java.lang.reflect.Method.invoke(Method.java:498) [?:1.8.0_332]     Any idea what 'java.lang.IllegalArgumentException: the given review item is not valid.' is really being caused by in this case?

            Asset Management added a comment - - edited

            Ran into a bug related to what I believe is the assumption by the Crucible migration plugin that Permission Scheme IDs will align (which will not happen often at all if migrating between 2 mature, well-used Crucible instances.)

            This is using Fisheye/Crucible version 4.8.9 Build:20220303135339 2022-03-03 13:56 as well as the Project Transfer for Crucible plugin version 1.0.5. 

            On the source Fisheye/Crucible (FECRU), the project export's root project.xml in the export zip file specifies an ID of '6' for the permission scheme used on the source server: "...<permissionSchemeId>6</permissionSchemeId><permissionSchemeName>PROJECT_Permission_Scheme</permissionSchemeName>..."

            As per the notes for migration, it only states "target instance must contain a permission scheme used by the project", which implies having a same-named permission scheme on the target FECRU server: https://confluence.atlassian.com/crucible/project-transfer-for-crucible-981151699.html

            When you recreate 'PROJECT_Permission_Scheme' on the target server, the import still fails to create the project with:

            2022-07-07 19:31:47,580 ERROR [qtp2085745483-2442 ] com.atlassian.crucible.plugins.transfer.impl.ImportProjectJob ImportProjectJob-run - Failed to import project 'PRJ'
            java.lang.NullPointerException
                    at com.atlassian.crucible.spi.impl.DefaultProjectService.validatePermissionSchemeIdAndNameConsistency(DefaultProjectService.java:216) [fisheye.jar:?]

            After tinkering with this a bit, and looking at the method off the stacktrace, it seemed to imply that the issue was with the permission scheme name and/or ID, so I cracked open the migration .zip file and edited the project.xml at the root, changed the "<permissionSchemeId>6</permissionSchemeId>" to align with what the actual ID on the target FECRU server was, so "<permissionSchemeId>49</permissionSchemeId>" and then it worked and proceeded on to import the reviews.

            Like expecting custom field IDs to be the same between 2 Jira instances, expecting permission schemes in 2 separate FECRU servers to have the same underlying ID is not a good assumption, maybe the migration plugin should just perform a name search to see if the user re-created the same-named permission scheme as what was used on the source FECRU server and then use whatever ID is assigned to it instead of expecting the IDs to align.

            Asset Management added a comment - - edited Ran into a bug related to what I believe is the assumption by the Crucible migration plugin that Permission Scheme IDs will align (which will not happen often at all if migrating between 2 mature, well-used Crucible instances.) This is using Fisheye/Crucible version 4.8.9 Build:20220303135339 2022-03-03 13:56 as well as the Project Transfer for Crucible plugin version 1.0.5.  On the source Fisheye/Crucible (FECRU), the project export's root project.xml in the export zip file specifies an ID of '6' for the permission scheme used on the source server: " ...<permissionSchemeId>6</permissionSchemeId><permissionSchemeName>PROJECT_Permission_Scheme</permissionSchemeName>... " As per the notes for migration, it only states "target instance must contain a permission scheme used by the project", which implies having a same-named permission scheme on the target FECRU server: https://confluence.atlassian.com/crucible/project-transfer-for-crucible-981151699.html When you recreate 'PROJECT_Permission_Scheme' on the target server, the import still fails to create the project with: 2022-07-07 19:31:47,580 ERROR [qtp2085745483-2442 ] com.atlassian.crucible.plugins.transfer.impl.ImportProjectJob ImportProjectJob-run - Failed to import project 'PRJ' java.lang.NullPointerException         at com.atlassian.crucible.spi.impl.DefaultProjectService.validatePermissionSchemeIdAndNameConsistency(DefaultProjectService.java:216) [fisheye.jar:?] After tinkering with this a bit, and looking at the method off the stacktrace, it seemed to imply that the issue was with the permission scheme name and/or ID, so I cracked open the migration .zip file and edited the project.xml at the root, changed the " <permissionSchemeId>6</permissionSchemeId> " to align with what the actual ID on the target FECRU server was, so " <permissionSchemeId>49</permissionSchemeId>"  and then it worked and proceeded on to import the reviews. Like expecting custom field IDs to be the same between 2 Jira instances, expecting permission schemes in 2 separate FECRU servers to have the same underlying ID is not a good assumption, maybe the migration plugin should just perform a name search to see if the user re-created the same-named permission scheme as what was used on the source FECRU server and then use whatever ID is assigned to it instead of expecting the IDs to align.

            Marek Parfianowicz added a comment - - edited

            > Perhaps this is a bug/limitation in Fisheye/Crucible itself, that for whatever reason Inactive users (from Crowd) don't show up in the User Browser at all.

            This is by design Indeed, Crucible doesn't show deleted users on the user list. You can see deleted users only in activities they participated in (e.g. commented by "Alice (Deleted user)" in a code review).

            > I tried creating a user inside of Fisheye/Crucible but because the user already exists in the Crowd Directory.

            You might try to use an Internal Directory to do the trick. If the internal directory has a higher priority than crowd (on top of the list), then you shall be able to add a user to this directory. It might require disabling Crowd for a moment of adding a user (so make sure you're logged in via administration password and not as an admin user). Once added, you can enable Crowd.

            Marek Parfianowicz added a comment - - edited > Perhaps this is a bug/limitation in Fisheye/Crucible itself, that for whatever reason Inactive users (from Crowd) don't show up in the User Browser at all. This is by design Indeed, Crucible doesn't show deleted users on the user list. You can see deleted users only in activities they participated in (e.g. commented by "Alice (Deleted user)" in a code review). > I tried creating a user inside of Fisheye/Crucible but because the user already exists in the Crowd Directory. You might try to use an Internal Directory to do the trick. If the internal directory has a higher priority than crowd (on top of the list), then you shall be able to add a user to this directory. It might require disabling Crowd for a moment of adding a user (so make sure you're logged in via administration password and not as an admin user). Once added, you can enable Crowd.

            Jeffrey Thornsen added a comment - - edited

            Thanks so much for the prompt reply,

            I tried creating a user inside of Fisheye/Crucible but because the user already exists in the Crowd Directory (albeit in an Inactive/Disabled state), Fisheye/Crucible won't let me create a new user with the same username.

            Perhaps this is a bug/limitation in Fisheye/Crucible itself, that for whatever reason Inactive users (from Crowd) don't show up in the User Browser at all, even though the user is in the cwd_user table in the underlying database.  I'm pretty sure other Atlassian applications (JIRA, Confluence, etc) don't work this way and list all of the Inactive users in their own UI.

            For example, in Jira I can create a user with the same username in multiple directories (E.g. Jira internal directory, Crowd) and the user that actually gets used is based on what order the Directories are listed in.  It seems I can't do the same thing in Fisheye/Crucible.

             

            I'm already doing edits on the XML files, so I guess the option of last resort is to do what you suggested and replace all the problematic usernames in the XML with fake/alternate accounts to get the data to import.

            Jeffrey Thornsen added a comment - - edited Thanks so much for the prompt reply, I tried creating a user inside of Fisheye/Crucible but because the user already exists in the Crowd Directory (albeit in an Inactive/Disabled state), Fisheye/Crucible won't let me create a new user with the same username. Perhaps this is a bug/limitation in Fisheye/Crucible itself, that for whatever reason Inactive users (from Crowd) don't show up in the User Browser at all, even though the user is in the cwd_user table in the underlying database.  I'm pretty sure other Atlassian applications (JIRA, Confluence, etc) don't work this way and list all of the Inactive users in their own UI. For example, in Jira I can create a user with the same username in multiple directories (E.g. Jira internal directory, Crowd) and the user that actually gets used is based on what order the Directories are listed in.  It seems I can't do the same thing in Fisheye/Crucible.   I'm already doing edits on the XML files, so I guess the option of last resort is to do what you suggested and replace all the problematic usernames in the XML with fake/alternate accounts to get the data to import.

            Hi Jeffrey,

            I couldn't find any other way to request support/help for this plugin.

            You can either post comments here or on community.atlassian.com.

            Do all the users in the backup XML file need to be Active/Enabled in order to import the backup? 

            In general yes, a target server should contain all users from a source server (or at least - all authors, moderators, review participants etc).

            Is there anything I can do to workaround this problem?

            If you cannot re-enable the problematic user then maybe you could create an artificial account for deleted users? Such account should be active (i.e. granted Fisheye&Crucible user permission). Then you could edit *.xml files in the export zip and replace all occurrences of this problematic username with the new one.

            Cheers
            Marek

            Marek Parfianowicz added a comment - Hi Jeffrey, I couldn't find any other way to request support/help for this plugin. You can either post comments here or on community.atlassian.com. Do all the users in the backup XML file need to be Active/Enabled in order to import the backup?  In general yes, a target server should contain all users from a source server (or at least - all authors, moderators, review participants etc). Is there anything I can do to workaround this problem? If you cannot re-enable the problematic user then maybe you could create an artificial account for deleted users? Such account should be active (i.e. granted Fisheye&Crucible user permission). Then you could edit *.xml files in the export zip and replace all occurrences of this problematic username with the new one. Cheers Marek

            Jeffrey Thornsen added a comment - - edited

            Hi, I couldn't find any other way to request support/help for this plugin.

            I'm trying to import a backup, but I keep getting an error related to Inactive/Disabled users.  The specific error in the UI is:

            • The user named '<username>' is not a Crucible user.

            The log on the server shows a giant Java stacktrace which begins with:

            • com.atlassian.crucible.plugins.transfer.impl.ImportProjectJob ImportProjectJob-run - Failed to import project '<PROJECT-KEY>'
            • com.atlassian.crucible.spi.services.NotFoundException: The user named '<username>' is not a Crucible user
            • at com.atlassian.crucible.spi.impl.DefaultReviewService.getLicensedUser(DefaultReviewService.java:2439) [fisheye.jar:?]

            We use Crowd (backed by Active Directory / LDAP) as the user directory.  The user being mentioned is in our Crowd server, but is Disabled/Inactive.  The user does not show up in the Fisheye/Crucible user browser in the web Interface, but if I query the underlying database the user is in the cwd_user table, just with the flag Active='F' (as expected).

            Do all the users in the backup XML file need to be Active/Enabled in order to import the backup?  Is there anything I can do to workaround this problem?

            Unfortunately, it's not possible for me to re-enable the affected users (even temporarily) due to our underlying Crowd/LDAP implementation.

            Jeffrey Thornsen added a comment - - edited Hi, I couldn't find any other way to request support/help for this plugin. I'm trying to import a backup, but I keep getting an error related to Inactive/Disabled users.  The specific error in the UI is: The user named '<username>' is not a Crucible user. The log on the server shows a giant Java stacktrace which begins with: com.atlassian.crucible.plugins.transfer.impl.ImportProjectJob ImportProjectJob-run - Failed to import project '<PROJECT-KEY>' com.atlassian.crucible.spi.services.NotFoundException: The user named '<username>' is not a Crucible user at com.atlassian.crucible.spi.impl.DefaultReviewService.getLicensedUser(DefaultReviewService.java:2439) [fisheye.jar:?] We use Crowd (backed by Active Directory / LDAP) as the user directory.  The user being mentioned is in our Crowd server, but is Disabled/Inactive.  The user does not show up in the Fisheye/Crucible user browser in the web Interface, but if I query the underlying database the user is in the cwd_user table, just with the flag Active='F' (as expected). Do all the users in the backup XML file need to be Active/Enabled in order to import the backup?  Is there anything I can do to workaround this problem? Unfortunately, it's not possible for me to re-enable the affected users (even temporarily) due to our underlying Crowd/LDAP implementation.

            when i click on the export/import link the page never loads.

            How many projects do you have?

            Marek Parfianowicz added a comment - when i click on the export/import link the page never loads. How many projects do you have?

            Yes, it has REST API, we didn't publish it yet as it's a beta version:

            POST /import multipart/form-data -> application/json
            GET /import/status  application/json -> application/json
            POST /export url-encoded -> application/json
            GET /export/status - application/json -> application/json
            GET /export/recent - application/json -> application/json
            DELETE /export/<projectKey>/delete  - application/json
            GET /export/<projectKey>/download  - application/json -> octet-stream
            

            Marek Parfianowicz added a comment - Yes, it has REST API, we didn't publish it yet as it's a beta version: POST /import multipart/form-data -> application/json GET /import/status application/json -> application/json POST /export url-encoded -> application/json GET /export/status - application/json -> application/json GET /export/recent - application/json -> application/json DELETE /export/<projectKey>/delete - application/json GET /export/<projectKey>/download - application/json -> octet-stream

            is there an API with the plugin - i have a LARGE number of projects, when i click on the export/import link the page never loads.

            Dana Jansen added a comment - is there an API with the plugin - i have a LARGE number of projects, when i click on the export/import link the page never loads.

              Unassigned Unassigned
              pkamal Partha
              Votes:
              105 Vote for this issue
              Watchers:
              88 Start watching this issue

                Created:
                Updated: