ClusterManager reports node as Active when node is down

Issue Summary

When a node is suddenly terminated (ie not shutdown gracefully), ClusterManager API still return the terminated node as active

Background

From the app developer

I have 2 node cluster of JIRA 7.9.0 (node1, node2). Each node is a docker. When I turn off node2 by stopping the docker container the other node still sees second node as active.

I've checked that in clusternode table node2 is active. I've checked that in clusternodeheartbeat table beat data are not updated when node2 is down.

The way I am getting node data:

clusterManager.findLiveNodes();
return clusterManager.getAllNodes();

When instead of killing node2 I shut it down gracefully (/etc/init.d/jira stop), then node2 status in clusternode table is updated to OFFLINE.

Jira should make use of clusternodehearbeat table to determine whether node is OFFLINE

Tables data after node2 was terminated - not gracefully
2.1) clusternode

node_id,node_state,timestamp,ip,cache_listener_port,node_build_number,node_version
node1,ACTIVE,1559562446911,30b2e7efd175,40001,79000,7.9.0
node2,ACTIVE,1559834524044,bb7ae1186aa1,40001,79000,7.9.0

2.2) clusternodeheartbeat

node_id,heartbeat_time,database_time
node1,1560462378461,1560462378462
node2,1559852296782,1559852296782

3) I killed a note to simulate a failure in DC cluster. In my case I stopped a docker on which Tomcat with Jira node was placed.

Environment

7.9.0

Steps to Reproduce

1. Have 2 nodes
2. Terminate node 2 instead of shutting down gracefully

Expected Results

When node2 is down other nodes should see it as offline.

Actual Results

When node2 is down other nodes see it as online.

Notes

• I haven't personally replicated this. I'm moving this in this project for more expertise

Workaround

Currently there is no known workaround for this behavior. A workaround will be added here when available