Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-74882

Restarting Jira 9 after upgrading from Jira 8 triggers a full reindex regardless of Reindex configs

XMLWordPrintable

    • 9
    • 6
    • Severity 1 - Critical
    • 45
    • Hide

      Resolution notes

      • Add these two JVM flags to the startup parameters:
        -Dcom.atlassian.jira.startup.allow.full.reindex=false
        -Dcom.atlassian.jira.startup.rebuild.local.index=false
        

      They can be added to the setenv.sh/bat in the "JVM_SUPPORT_RECOMMENDED_ARGS" variable.

      • Also keep-or-add this line to the jira-config.properties:
        upgrade.reindex.allowed=false
        

      Follow-up Documentation updates

      We recently published a KB article with a clear mention to these configs and requested updates to the current upgrade guides (to merge content):

      Show
      Resolution notes Add these two JVM flags to the startup parameters: -Dcom.atlassian.jira.startup.allow.full.reindex=false -Dcom.atlassian.jira.startup.rebuild.local.index=false They can be added to the setenv.sh/bat in the "JVM_SUPPORT_RECOMMENDED_ARGS" variable. Also keep-or-add this line to the jira-config.properties: upgrade.reindex.allowed=false Follow-up Documentation updates We recently published a KB article with a clear mention to these configs and requested updates to the current upgrade guides (to merge content): Practical guide on upgrading Jira Data Center to version 9 or 9.4 LTS JRASERVER-74957 — Review and update the Jira 9 upgrade guides.

      Issue Summary

      When Jira is upgraded from 8.x to 9.x and the first node starts up for the first time, it performs the database upgrade and adheres to the jira-config.properties config:

      upgrade.reindex.allowed=false
      

      However, if the node's restarted, this second startup kicks-off a full reindex regardless.
       

      Impact

      Restarting Jira right after upgrade is not uncommon and reasons, vary like:

      • Admins realizing some customizations to config files were not re-applied
      • Admins realizing JVM custom parameters were not re-applied
      • Any other issue that didn't prevent DB upgrade tasks but called for a restart

      Jira 8 behaved in a way the "upgrade.reindex.allowed=false" would bypass full reindexes on every restart, but Jira 9 enforces a healthy Index to start-up properly (either local or from the shared home, or else it full reindexes).

      It seems Jira 9 interprets the config differently (maybe correctly now, and it was Jira 8 that had a "bug feature"): "upgrade.reindex.allowed" is only relevant on the startup that's actually performing the upgrade. A restart right after the upgrade's finished is no longer considered "an upgrade" and the config is thus ignored.

      The call for restarts may be unpredictable and may happen in one environment more than in another, and this full reindex is getting admins off guard and compromising the maintenance schedules and outage windows, leading to unplanned downtime.
       

      Impact aggravation

      An unexpected reindex is no good, but the impact's much worsened by Jira 9 taking much longer to reindex than Jira 8 on some datasets: JRASERVER-74787.
       

      Steps to Reproduce

      1. Install Jira 8.20 (2 node cluster)
      2. Load some data into it (through sample projects or Data Generator for Jira)
      3. Perform a full reindex
      4. Bring both nodes down
      5. Upgrade Jira to 9.4 in "node1" with "upgrade.reindex.allowed=false"
      6. Confirm in the logs no full reindex was kicked-off
      7. Restart Jira
      8. Observe the logs for full reindex messages and that Jira's locked while reindexing
        (with just a few data the outage may not be perceptible thus the logs are the main evidence of full reindex)
        2023-02-22 13:01:25,586-0300 JiraTaskExecutionThread-1 INFO anonymous     [c.a.j.util.index.CompositeIndexLifecycleManager] Reindex All COMPLETED without any errors. Total time: 4683ms. Reindex run: 4
        

      Expected Results

      Jira would refrain from kicking-off an automatic reindex and wait for the admin to start it manually through the UI (or REST API).
       

      Actual Results

      The second startup after the upgrade routines enforce a Jira 9 compatible index. If it can't find one locally, it tries for the shared-home and last, locks itself and performs a full reindex.
       

      Workaround

      -Dcom.atlassian.jira.startup.allow.full.reindex=false
      -Dcom.atlassian.jira.startup.rebuild.local.index=false
      

      They can be added to the setenv.sh/bat in the "JVM_SUPPORT_RECOMMENDED_ARGS" variable.

      We also published a Practical guide on upgrading Jira Data Center to version 9 or 9.4 LTS with a clear mention to these configs and requested other documentation updates through JRASERVER-74957

      (updated on March 14th 2023)

              rmartinez3@atlassian.com Rodrigo Martinez
              rmartinez3@atlassian.com Rodrigo Martinez
              Votes:
              3 Vote for this issue
              Watchers:
              20 Start watching this issue

                Created:
                Updated:
                Resolved: