Uploaded image for project: 'Jira Service Management Data Center'
  1. Jira Service Management Data Center
  2. JSDSERVER-8451

Longer time needed for imports in Insight version 8.6.12 and above

XMLWordPrintable

      Atlassian Update – 17 March 2022

      Hi everyone,

      We are pleased to let you know that a bug fix for the import performance regression is now available in Jira Service Management Data Center versions 4.20.7, 4.22.1 and the Insight Marketplace app version 8.9.10.

      This performance fix increases the import speed when there are object type hierarchies in the schemas. We’ve seen some significant improvements and look forward to having customers put it to use!

      We're initially releasing the fix behind a feature flag for early adopters to experiment with. We’re ready to provide a rollback for you if anything unexpected occurs. Follow the instructions below to enable the flag.

      Kind regards,

      Craig Shannon

      Jira Service Management, Data Center & Server

      Instructions to enable the feature flags in 4.20.7 / 4.22.1

      There are two feature flags which will turn on this new behavior:

      insight.index.object.type.children

      insight.index.object.type.reindex.on.failure

      To activate them, see Enable Dark Feature in Jira.

      Once you enable those feature flags, you need to run a reindex as the object type structure in the index changes to improve the lookup performance of the object type hierarchy. You can do it from <JIRA URL>/secure/admin/InsightPluginShowIndex.jspa. Make sure to do a clean reindex to ensure that the old object type index file on disk is removed.

      If you encounter any issues, disabling the feature flags will revert back the behavior and there is no need to perform a reindex again.

      Issue Summary

      Much longer time needed for imports in Insight version 8.6.12 and above. This also causes CPU spike up to 99% and the import process may seem halted on the UI.

      Steps to Reproduce

      1. With Insight 8.6.11, set up and import from the CVE-Database as per this documentation. For testing purpose, we choose CVE 2020.
      2. Note the time taken to complete the import.
      3. Upgrade to Insight 8.6.12 or any newer version, create another schema and perform the exact same import.
      4. Note the time taken to complete the import and the CPU usage.

      Expected Results

      The time taken to complete the import in both versions is almost the same

      Actual Results

      • It takes much longer to complete the import in Insight version 8.6.12 and above. Based on the comparison of the logs, we can see that the step to decide actions for all objects is taking more time in the newer Insight versions (103ms vs 401608ms).
      • CPU spike can also be observed along the import process especially on step 3 (decide actions for all objects) and as soon as the import is done, CPU usage is back to normal. Due to the much longer process, the impact of CPU spike is amplified.
        In some cases, the import seems halted for hours on step 3 and CPU usage is constantly at its peak. The import has to be terminated manually by canceling the import, disabling and re-enabling Insight or restart Jira.

        Insight 8.6.11

        2021-07-08 13:47:39,598 [insight-InsightThreadGroup-worker-thread-0] | NVD>>> Mapped all data to internal objects in 62805 ms
2021-07-08 13:47:39,702 [insight-InsightThreadGroup-worker-thread-0] | NVD>>> Decided actions for all objects in 103 ms
2021-07-08 13:47:39,702 [insight-InsightThreadGroup-worker-thread-0] | NVD>>> Objects to import 38801
...
2021-07-08 13:55:50,762 [insight-InsightThreadGroup-worker-thread-0] | Handled 32200 objects with outbound references
2021-07-08 13:55:50,763 [insight-InsightThreadGroup-worker-thread-0] | Finished updating object references in 250704 ms
2021-07-08 13:55:50,763 [insight-InsightThreadGroup-worker-thread-0] | NVD>>> Finished import with id 18 of type insight-cve-import at time 2021-07-08T13:55:50.763233. Import took 660570 ms

      Insight 8.6.12 and above

      2021-07-08 11:26:05,055 [insight-InsightThreadGroup-worker-thread-4] | NVD>>> Mapped all data to internal objects in 59958 ms
2021-07-08 11:32:46,664 [insight-InsightThreadGroup-worker-thread-4] | NVD>>> Decided actions for all objects in 401608 ms
2021-07-08 11:32:46,664 [insight-InsightThreadGroup-worker-thread-4] | NVD>>> Objects to import 38254
...
2021-07-08 11:40:27,822 [insight-InsightThreadGroup-worker-thread-4] | Handled 31326 objects with outbound references
2021-07-08 11:40:27,823 [insight-InsightThreadGroup-worker-thread-4] | Finished updating object references in 242887 ms
2021-07-08 11:40:27,823 [insight-InsightThreadGroup-worker-thread-4] | NVD>>> Finished import with id 17 of type insight-cve-import at time 2021-07-08T11:40:27.823297. Import took 1040906 ms

      No blocked threads but attached threads below are often accompanied with high CPU usage.

      Workaround

      Currently there is no known workaround for this behavior. A workaround will be added here when available

        1. AttributeValuePredicateBase.txt
          4 kB
          Michelle Chin
        2. InsightIndexBase.txt
          3 kB
          Michelle Chin
        3. Screen Shot 2022-03-23 at 7.53.03 pm.png
          33 kB
          Craig Shannon
        4. Screen Shot 2022-03-23 at 8.29.33 pm.png
          504 kB
          Craig Shannon

            [JSDSERVER-8451] Longer time needed for imports in Insight version 8.6.12 and above

            Jorden Van Bogaert added a comment -

            Thanks for checking! It seems that I indeed forgot the .enabled part of the flag. After enabling, it was much faster to read external data and seemed to jump much further in the "deciding actions for objects", however we might still be impacted by another issue. At some point in the "deciding actions", it starts going very slow (3-4 objects per second, while we have 200.000+ in the schema, it's the Jira configuration importer add-on).

            I'll create a separate support ticket for that.

            Thanks for the help!

            Jorden Van Bogaert added a comment - Thanks for checking! It seems that I indeed forgot the .enabled part of the flag. After enabling, it was much faster to read external data and seemed to jump much further in the "deciding actions for objects", however we might still be impacted by another issue. At some point in the "deciding actions", it starts going very slow (3-4 objects per second, while we have 200.000+ in the schema, it's the Jira configuration importer add-on). I'll create a separate support ticket for that. Thanks for the help!

            Craig Shannon added a comment - - edited

            Hi bc4ae08bb47a,

            I done some verification on the marketplace app 8.9.10 to ensure the expected fix that we implemented is there, which tackles a regression that was added where there are object type hierarchies. I have attached a screenshot with the results where query times go from over 8 seconds to sub second on a setup that triggers this regression.

            However, you could be encountering a different issue from the one that we have solved in this ticket. Let us know how you get on with the feature flag to verify you have 'enabled' appended at the end.

            Thanks,
            Craig.

            Craig Shannon added a comment - - edited Hi bc4ae08bb47a , I done some verification on the marketplace app 8.9.10 to ensure the expected fix that we implemented is there, which tackles a regression that was added where there are object type hierarchies. I have attached a screenshot with the results where query times go from over 8 seconds to sub second on a setup that triggers this regression. However, you could be encountering a different issue from the one that we have solved in this ticket. Let us know how you get on with the feature flag to verify you have 'enabled' appended at the end. Thanks, Craig.

            Craig Shannon added a comment - - edited

            Hi bc4ae08bb47a,

            Sorry to hear this isn't working as expected.

            The screenshot didn't come through, however I just want to confirm that for the feature flag, you have added `insight.index.object.type.children.enabled` and `insight.index.object.type.reindex.on.failure.enabled` on the dark features configuration page? I have added a screenshot as an example.

            If this is the case, we are likely best to follow up via a support ticket where we can assist with capturing a support zip and some thread dumps to see what is consuming the CPU.

            Thanks,
            Craig.

            Craig Shannon added a comment - - edited Hi bc4ae08bb47a , Sorry to hear this isn't working as expected. The screenshot didn't come through, however I just want to confirm that for the feature flag, you have added `insight.index.object.type.children.enabled` and `insight.index.object.type.reindex.on.failure.enabled` on the dark features configuration page? I have added a screenshot as an example. If this is the case, we are likely best to follow up via a support ticket where we can assist with capturing a support zip and some thread dumps to see what is consuming the CPU. Thanks, Craig.

            Jorden Van Bogaert added a comment -

            For us this is not fixed with the app version 8.9.10 (I have the features flags configured and did a re-index before running the import).

            As per below graph, you can see the import starts around 1AM. Around 8AM I shutdown the Jira server as the import is still running. In the past, the same import took around 40 minutes, but I never recalled it spiking the CPU to 100% (until it broke with a new version from insight).

            Jorden Van Bogaert added a comment - For us this is not fixed with the app version 8.9.10 (I have the features flags configured and did a re-index before running the import). As per below graph, you can see the import starts around 1AM. Around 8AM I shutdown the Jira server as the import is still running. In the past, the same import took around 40 minutes, but I never recalled it spiking the CPU to 100% (until it broke with a new version from insight).

            Craig Shannon added a comment -

            Hi bc4ae08bb47a,

            Thanks for reaching out. The steps to enable for the app version 8.9.10 will be the same, it was just that the docs linked to the later Jira version (8.22), but enabling dark features hasn't changed, so the steps will be the same. I'll update my comment to make this clearer.

            Thanks,
            Craig.

            Craig Shannon added a comment - Hi bc4ae08bb47a , Thanks for reaching out. The steps to enable for the app version 8.9.10 will be the same, it was just that the docs linked to the later Jira version (8.22), but enabling dark features hasn't changed, so the steps will be the same. I'll update my comment to make this clearer. Thanks, Craig.

            Jorden Van Bogaert added a comment -

            Hi 01656e440512

            You mention "Instructions to enable the feature flags in 4.20.7 / 4.22.1", does that mean that the app version 8.9.10 (for server users still) automatically has the fix applied without enabling any feature flag? Or are there specific steps to take for it too?

            Kind regards
            Jorden

            Jorden Van Bogaert added a comment - Hi  01656e440512 You mention "Instructions to enable the feature flags in 4.20.7 / 4.22.1", does that mean that the app version 8.9.10 (for server users still) automatically has the fix applied without enabling any feature flag? Or are there specific steps to take for it too? Kind regards Jorden

            Craig Shannon added a comment - - edited
            Atlassian Update – 17 March 2022

            Hi everyone,

            We are pleased to let you know that a bug fix for the import performance regression is now available in Jira Service Management Data Center versions 4.20.7, 4.22.1 and the Insight Marketplace app version 8.9.10.

            This performance fix increases the import speed when there are object type hierarchies in the schemas. We’ve seen some significant improvements and look forward to having customers put it to use!

            We're initially releasing the fix behind a feature flag for early adopters to experiment with. We’re ready to provide a rollback for you if anything unexpected occurs. Follow the instructions below to enable the flag.

            Kind regards,

            Craig Shannon

            Jira Service Management, Data Center & Server

            Instructions to enable the feature flags in Jira

            There are two feature flags which will turn on this new behavior:

            insight.index.object.type.children

            insight.index.object.type.reindex.on.failure

            To activate them, see Enable Dark Feature in Jira.

            Once you enable those feature flags, you need to run a reindex as the object type structure in the index changes to improve the lookup performance of the object type hierarchy. You can do it from <JIRA URL>/secure/admin/InsightPluginShowIndex.jspa. Make sure to do a clean reindex to ensure that the old object type index file on disk is removed.

            If you encounter any issues, disabling the feature flags will revert back the behavior and there is no need to perform a reindex again.

            Craig Shannon added a comment - - edited Atlassian Update – 17 March 2022 Hi everyone, We are pleased to let you know that a bug fix for the import performance regression is now available in Jira Service Management Data Center versions 4.20.7, 4.22.1 and the Insight Marketplace app version 8.9.10. This performance fix increases the import speed when there are object type hierarchies in the schemas. We’ve seen some significant improvements and look forward to having customers put it to use! We're initially releasing the fix behind a feature flag for early adopters to experiment with. We’re ready to provide a rollback for you if anything unexpected occurs. Follow the instructions below to enable the flag. Kind regards, Craig Shannon Jira Service Management, Data Center & Server Instructions to enable the feature flags in Jira There are two feature flags which will turn on this new behavior: insight.index.object.type.children insight.index.object.type.reindex.on.failure To activate them, see Enable Dark Feature in Jira . Once you enable those feature flags, you need to run a reindex as the object type structure in the index changes to improve the lookup performance of the object type hierarchy. You can do it from <JIRA URL>/secure/admin/InsightPluginShowIndex.jspa. Make sure to do a clean reindex to ensure that the old object type index file on disk is removed. If you encounter any issues, disabling the feature flags will revert back the behavior and there is no need to perform a reindex again.

            Charlie Marriott added a comment -
            Atlassian Update – 22 February 2022

            Hi everyone,

            Thank you for your interest in this issue.

            After investigating this bug, we have identified the cause of the import performance regression and have implemented a solution. We are currently in the final stages of testing the fix to ensure the expected performance improvement is achieved.

            If everything goes according to plan, we will be backporting this fix to Long Term Support versions. It will also be available in the next Jira Service Management release.

            Please continue to follow the status of this ticket for further updates and fix versions when available.

            Kind regards,

            Charlie

            Jira Service Management, Data Center & Server

            Charlie Marriott added a comment - Atlassian Update – 22 February 2022 Hi everyone, Thank you for your interest in this issue. After investigating this bug, we have identified the cause of the import performance regression and have implemented a solution. We are currently in the final stages of testing the fix to ensure the expected performance improvement is achieved. If everything goes according to plan, we will be backporting this fix to Long Term Support versions. It will also be available in the next Jira Service Management release. Please continue to follow the status of this ticket for further updates and fix versions when available. Kind regards, Charlie Jira Service Management, Data Center & Server

            Fabrizio Galletti (Getconnected) added a comment -

            .

            Fabrizio Galletti (Getconnected) added a comment - .

            JIRA Supporto added a comment -

            @michin @Atlassian any news?

            JIRA Supporto added a comment - @ michin @Atlassian any news?

              Unassigned Unassigned
              michin Michelle Chin
              Affected customers:
              35 This affects my team
              Watchers:
              45 Start watching this issue

                Created:
                Updated:
                Resolved: