Resolution: Not a bug
9.0.0, 9.4.0, 9.4.3, 9.7.0
Severity 1 - Critical
Resolution notes Add these two JVM flags to the startup parameters: -Dcom.atlassian.jira.startup.allow.full.reindex=false -Dcom.atlassian.jira.startup.rebuild.local.index=false They can be added to the setenv.sh/bat in the "JVM_SUPPORT_RECOMMENDED_ARGS" variable. Also keep-or-add this line to the jira-config.properties: upgrade.reindex.allowed=false Follow-up Documentation updates We recently published a KB article with a clear mention to these configs and requested updates to the current upgrade guides (to merge content): Practical guide on upgrading Jira Data Center to version 9 or 9.4 LTS JRASERVER-74957 — Review and update the Jira 9 upgrade guides.
When Jira is upgraded from 8.x to 9.x and the first node starts up for the first time, it performs the database upgrade and adheres to the jira-config.properties config:
However, if the node's restarted, this second startup kicks-off a full reindex regardless.
Restarting Jira right after upgrade is not uncommon and reasons, vary like:
- Admins realizing some customizations to config files were not re-applied
- Admins realizing JVM custom parameters were not re-applied
- Any other issue that didn't prevent DB upgrade tasks but called for a restart
Jira 8 behaved in a way the "upgrade.reindex.allowed=false" would bypass full reindexes on every restart, but Jira 9 enforces a healthy Index to start-up properly (either local or from the shared home, or else it full reindexes).
It seems Jira 9 interprets the config differently (maybe correctly now, and it was Jira 8 that had a "bug feature"): "upgrade.reindex.allowed" is only relevant on the startup that's actually performing the upgrade. A restart right after the upgrade's finished is no longer considered "an upgrade" and the config is thus ignored.
The call for restarts may be unpredictable and may happen in one environment more than in another, and this full reindex is getting admins off guard and compromising the maintenance schedules and outage windows, leading to unplanned downtime.
An unexpected reindex is no good, but the impact's much worsened by Jira 9 taking much longer to reindex than Jira 8 on some datasets:
- Install Jira 8.20 (2 node cluster)
- Load some data into it (through sample projects or Data Generator for Jira)
- Perform a full reindex
- Bring both nodes down
- Upgrade Jira to 9.4 in "node1" with "upgrade.reindex.allowed=false"
- Confirm in the logs no full reindex was kicked-off
- Restart Jira
- Observe the logs for full reindex messages and that Jira's locked while reindexing
(with just a few data the outage may not be perceptible thus the logs are the main evidence of full reindex)
Jira would refrain from kicking-off an automatic reindex and wait for the admin to start it manually through the UI (or REST API).
The second startup after the upgrade routines enforce a Jira 9 compatible index. If it can't find one locally, it tries for the shared-home and last, locks itself and performs a full reindex.