Status: Closed (View Workflow)
6.3.15, 6.4.14, 7.3.7, 7.4.4, 7.2.11, 7.2.13, 7.6.4, 7.10.0, 7.11.2, 7.13.1, 7.6.11
Severity 2 - Major
Atlassian Update - 6 June 2019 Dear Jira users, we're glad to announce that this issue will be addressed in our upcoming 8.3.0 release! Jira DC will now recognize when a re-indexing node goes down and leaves a stale reindexing job in progress. Admins will be notified on the Re-Indexing page and prompted to remove the job, clearing the way to run future re-indexes. We plan to back port this fix to version 7.13 (our latest Enterprise Release version). Please watch the fix versions in this issue for further updates. This fix will not be backported to the 7.6 series; however, a couple of workarounds are available. Please see the Problem mitigation section under the Description of this issue. Thank you, Grazyna Kaszkur Product Manager, Jira Server and Data Center
When a re-indexing node goes down (terminated abruptly or crashed), all the other nodes are not able to tell that re-indexing is no longer in progress. There is no impact for normal JIRA functionality. This is UI problem and you will be not able to run reindex.
Please note that there is another similar problem related to project reindex, see JRASERVER-72055.
- JIRA Data Center
- Apache Load Balancer
- Setup a Jira Data Center with atleast 2 nodes
- Start re-indexing from one node
- Access the re-indexing administration page from the other nodes to see the Re-indexing progress summary
- Abruptly terminate the JIRA process on the re-indexing node
- Access re-indexing administration again on the node that is still online
This should show that the re-indexing process has been terminated, and should allow you to restart re-indexing if needed.
The online node(s) are not able to detect that the re-indexing started in the other node has been terminated.
You will see the error in UI: 'Lock JIRA and rebuild index' option is unavailable while other indexing operations are in progress.'
- When user starts FullReindexing JIRA schedules the job at specific node and add non-cancellable task into Global cache. This cache is synced with each node.
- When node dies, value still present in the cache and all nodes see it and believes that index is still running.
- Whole cluster shutdown is needed to flush the cache
- Looking into the database from our test we can see that there is an entry in the replicatedindexoperation table such as the following for each re-index triggered:
- There is another entry when this reindex ends. For the killed re-indexing node there is a start with no end entry.
To partially address the problem and mitigate some of the possible scenarios, we implemented 2 new features:
JRASERVER-68885- Remove the stale indexing Job associated with the current node on startup JRASERVER-68616- As an JIRA Datacenter Administrator I want to delete the reindexing task from offline node
Please check them for more details.
- Shut down all nodes in the cluster.
- This is needed since JIRA keeps cluster reindex status in memory (see Notes section). Leaving one available and restarting all others will keep the in-memory cache.
- Start nodes