Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-74787

Full reindex operation slow down in Jira 9.0.0 caused by fetching issue versions from database in a single thread

XMLWordPrintable

    • 9
    • 39
    • Severity 2 - Major
    • 547
    • Hide

      Atlassian Update – 18 May 2023

      Dear Customers,

      We’d like to announce that the fix for this issue has been released in the 9.4.6 update version.

      Please note that this fix addresses a specific cause of decreased performance that had impacted many of our customers.

      If you still experience significant slowdowns in full reindexing on the 9.4.6 release, please contact support and provide a detailed description of the issue. Please note that, since the 9.0 release, there has been an improved index consistency guarantee. As a result, it is expected that there will be an increase in full reindex duration of no more than 3 minutes per 1 million issues. For more information on this, please refer to the Important Note below.

      Furthermore, we would like to draw your attention to the attached instruction, Confirming the fix will help before the upgrade which may help you understand if the root cause of the decreased performance is related to this bug.

       
      Related links of interest:

      Important Note

      Please be aware that the fix does not completely remove the increase of full reindex duration between versions 8.x and 9.x.

      Since Jira 9.0, there is an expected slight increase in full reindex duration compared to prior versions. The increase is due to the improved index consistency guarantee, ensuring a more reliable and accurate indexing process. As stated in the Release Notes, the expected increase in full reindex duration should not be larger than 3 minutes per 1 million issues.

      Confirming the fix will help before the upgrade

      For those who would like to check before upgrading if the increased indexing time they are experiencing will be resolved by the fix, we have attached a simple set of instructions which will guide you through verifying this in the logs (see below).

      We kindly request customers who, after following the attached instructions, find that their increased full reindexing time does not align with the known factors, to raise a new bug and contact Jira support for further assistance. Your feedback is invaluable, and we appreciate your cooperation in helping us improve our product.

      Instruction

      The goal of this steps is to gather information about how much time indexing process spent on retrieving versioning data(this operation happening, unnecessarily, in single-threaded part of full reindex is identified as the direct cause of the increased time to reindex) from database and confirm it’s the only cause of the bug. Because stats containing this info contain values accumulated from the moment of node start up, we need to subtract the value of the stat measuring this just before the reindex from the value just after the reindex.

      1. Run following command in Jira logs directory
        grep -R -e "Reindex All starting..." -e ".JIRA-STATS. .VERSIONING. total stats:" -e "ReindexAll took:" atlassian-jira.log* | less
      2. Locate most recent line containing Reindex All starting...
      3. Find previous occurrence of [JIRA-STATS] [VERSIONING] total stats: before the line from pt. 2.
      4. Note down the value of getIssueVersionMillis.sum as sum-start
      5. Locate most recent occurrence of line containing grep -R "ReindexAll took:" atlassian-jira.log* }}Note down the value as {{reindex-duration (the value is in milliseconds)
      6. Locate the next line containing [JIRA-STATS] [VERSIONING] total stats: after the line from pt. 5.
      7. Note down the value of getIssueVersionMillis.sum as sum-end
      8. Calculate getting-version-cost = sum-end - sum-start

      Checking if the fix will help before upgrade

      Check the difference between reindex-duration __ and __ getting-version-cost - should be roughly the same as the difference between full reindex on Jira 9.x and on Jira with version prior to 9.0. If the actual difference is significantly larger please contact Support as this might be caused by yet unknown bug not addressed by the current fix.

      Checking if the fix fully addressed the problem after the upgrade

      Check the difference between

      • reindex-duration and
      • getting-version-cost divided by number of  indexing threads (20 by default since 8.x)

      should be roughly the same as regression from 8.20 and be roughly equal to 3 minutes per million issues. In case:

      • the difference is significantly larger than getting-version-cost or
      • the difference is significantly larger than expected 3 minutes per million issues

      please contact Support, as this might be caused by yet unknown bug not addressed by this fix.

      Show
      Atlassian Update – 18 May 2023 Dear Customers, We’d like to announce that the fix for this issue has been released in the 9.4.6 update version. Please note that this fix addresses a specific cause of decreased performance that had impacted many of our customers. If you still experience significant slowdowns in full reindexing on the 9.4.6 release, please contact support and provide a detailed description of the issue. Please note that, since the 9.0 release, there has been an improved index consistency guarantee. As a result, it is expected that there will be an increase in full reindex duration of no more than 3 minutes per 1 million issues . For more information on this, please refer to the Important Note below. Furthermore, we would like to draw your attention to the attached instruction, Confirming the fix will help before the upgrade which may help you understand if the root cause of the decreased performance is related to this bug.   Related links of interest: Jira Data Center announcements Jira Software release notes Important Note Please be aware that the fix does not completely remove the increase of full reindex duration between versions 8.x and 9.x. Since Jira 9.0, there is an expected slight increase in full reindex duration compared to prior versions. The increase is due to the improved index consistency guarantee, ensuring a more reliable and accurate indexing process. As stated in the Release Notes , the expected increase in full reindex duration should not be larger than 3 minutes per 1 million issues . Confirming the fix will help before the upgrade For those who would like to check before upgrading if the increased indexing time they are experiencing will be resolved by the fix, we have attached a simple set of instructions which will guide you through verifying this in the logs (see below). We kindly request customers who, after following the attached instructions, find that their increased full reindexing time does not align with the known factors, to raise a new bug and contact Jira support for further assistance. Your feedback is invaluable, and we appreciate your cooperation in helping us improve our product. Instruction The goal of this steps is to gather information about how much time indexing process spent on retrieving versioning data(this operation happening, unnecessarily, in single-threaded part of full reindex is identified as the direct cause of the increased time to reindex) from database and confirm it’s the only cause of the bug. Because stats containing this info contain values accumulated from the moment of node start up, we need to subtract the value of the stat measuring this just before the reindex from the value just after the reindex. Run following command in Jira logs directory grep -R -e "Reindex All starting..." -e ".JIRA-STATS. .VERSIONING. total stats:" -e "ReindexAll took:" atlassian-jira.log* | less Locate most recent line containing Reindex All starting... Find previous occurrence of [JIRA-STATS] [VERSIONING] total stats: before the line from pt. 2. Note down the value of getIssueVersionMillis.sum as sum-start Locate most recent occurrence of line containing grep -R "ReindexAll took:" atlassian-jira.log* }}Note down the value as {{reindex-duration (the value is in milliseconds) Locate the next line containing [JIRA-STATS] [VERSIONING] total stats: after the line from pt. 5. Note down the value of getIssueVersionMillis.sum as sum-end Calculate getting-version-cost = sum-end - sum-start Checking if the fix will help before upgrade Check the difference between reindex-duration __ and __ getting-version-cost - should be roughly the same as the difference between full reindex on Jira 9.x and on Jira with version prior to 9.0. If the actual difference is significantly larger please contact Support as this might be caused by yet unknown bug not addressed by the current fix. Checking if the fix fully addressed the problem after the upgrade Check the difference between reindex-duration and getting-version-cost divided by number of  indexing threads (20 by default since 8.x) should be roughly the same as regression from 8.20 and be roughly equal to 3 minutes per million issues. In case: the difference is significantly larger than getting-version-cost or the difference is significantly larger than expected 3 minutes per million issues please contact Support, as this might be caused by yet unknown bug not addressed by this fix.

      Issue Summary

      The full reindex process has slowed down significantly for some customers. This is due to extra operations performed during reindexing that improve index correctness.

      Steps to Reproduce

      1. Update to Jira 9.x
      2. Run full re-index

      Expected Results

      There is no significant time degradation comparing to 8.20.x.

      Upgrading from 8.x to 9.x - Index changes

      Full reindex process durations might slightly increase(our tests estimate this increase can be up to 3 minutes per 1M issues)

      Actual Results

      Operation takes significantly more time.

      Jira Versioning stats show that total time spent by fetching(see the Instruction above on how to check that) is roughly equal to extra duration of reindexing when compared to duration on the instance before upgrading to 9.x.

      Workaround

      Currently, there is no known workaround for this behaviour. A workaround will be added here when available.

       

              755f87a38635 Krzysztof Krysiak
              kcichy Kamil Cichy (Inactive)
              Votes:
              35 Vote for this issue
              Watchers:
              113 Start watching this issue

                Created:
                Updated:
                Resolved: