-
Bug
-
Resolution: Fixed
-
High
-
6.3.15, 6.4.14, 7.3.7, 7.4.4, 7.2.11, 7.2.13, 7.6.4, 7.10.0, 7.11.2, 7.13.1, 7.6.11
-
6.03
-
143
-
Severity 2 - Major
-
688
-
-
Summary
When a re-indexing node goes down (terminated abruptly or crashed), all the other nodes are not able to tell that re-indexing is no longer in progress. There is no impact for normal JIRA functionality. This is UI problem and you will be not able to run reindex.
Please note that there is another similar problem related to project reindex, see JRASERVER-72055.
Environment
- JIRA Data Center
- Apache Load Balancer
Steps to Reproduce
- Setup a Jira Data Center with atleast 2 nodes
- Start re-indexing from one node
- Access the re-indexing administration page from the other nodes to see the Re-indexing progress summary
- Abruptly terminate the JIRA process on the re-indexing node
- Access re-indexing administration again on the node that is still online
Expected Results
This should show that the re-indexing process has been terminated, and should allow you to restart re-indexing if needed.
Actual Results
The online node(s) are not able to detect that the re-indexing started in the other node has been terminated.
You will see the error in UI: 'Lock JIRA and rebuild index' option is unavailable while other indexing operations are in progress.'
Notes
- When user starts FullReindexing JIRA schedules the job at specific node and add non-cancellable task into Global cache. This cache is synced with each node.
- When node dies, value still present in the cache and all nodes see it and believes that index is still running.
- Whole cluster shutdown is needed to flush the cache
- Looking into the database from our test we can see that there is an entry in the replicatedindexoperation table such as the following for each re-index triggered:
mysql> SELECT * FROM replicatedindexoperation; +-------+---------------------+---------+----------------+-------------+--------------+--------------------------+-------------------------+ | ID | INDEX_TIME | NODE_ID | AFFECTED_INDEX | ENTITY_TYPE | AFFECTED_IDS | OPERATION | FILENAME | +-------+---------------------+---------+----------------+-------------+--------------+--------------------------+-------------------------+ | 10500 | 2015-11-19 20:00:11 | tnode1 | ALL | NONE | | FULL_REINDEX_START | | | 10501 | 2015-11-19 20:32:02 | tnode1 | ALL | NONE | | FULL_REINDEX_END | IndexSnapshot_10500.zip | +-------+---------------------+---------+----------------+-------------+--------------+--------------------------+-------------------------+ 4 rows in set (0.00 sec)
- There is another entry when this reindex ends. For the killed re-indexing node there is a start with no end entry.
Problem mitigation
To partially address the problem and mitigate some of the possible scenarios, we implemented 2 new features:
JRASERVER-68885- Remove the stale indexing Job associated with the current node on startupJRASERVER-68616- As an JIRA Datacenter Administrator I want to delete the reindexing task from offline node
Please check them for more details.
Workarounds
- Shut down all nodes in the cluster.
- This is needed since JIRA keeps cluster reindex status in memory (see Notes section). Leaving one available and restarting all others will keep the in-memory cache.
- Start nodes
- causes
-
JRASERVER-66722 As an JIRA Datacenter Administrator I want to delete stuck global tasks
- Closed
-
JRASERVER-68616 As an JIRA Datacenter Administrator I want to delete reindexing task from offline node
- Closed
-
JRASERVER-68885 Remove the stale indexing Job associated with current node on startup
- Closed
-
HOT-85608 Loading...
-
HOT-86049 Loading...
-
ITOPS-1071 Loading...
-
ITOPS-1074 Loading...
-
ITOPS-1175 Loading...
- is caused by
-
JRASERVER-67138 /status url returns ERROR instead of MAINTENANCE during foreground reindexing
- Closed
- relates to
-
JRASERVER-66204 Bulk Operation can get stuck in JIRA Data Center
- Closed
-
JRASERVER-72055 Abruptly stopping node while project reindex is in progress causes inability to reindex Jira
- Gathering Impact
- resolves
-
JRASERVER-66969 Discourage Background Indexing for Datacenter instances in Indexing Page
- Closed
-
JRASERVER-68006 Add background reindex warning for large data center environment
- Closed
- Mentioned in
- mentioned in
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...
-
Page Loading...