Uploaded image for project: 'Jira Server and Data Center'
  1. Jira Server and Data Center
  2. JRASERVER-69493

ClusterManager reports node as Active when node is down

    XMLWordPrintable

Details

    Description

      Issue Summary

      When a node is suddenly terminated (ie not shutdown gracefully), ClusterManager API still return the terminated node as active

      Background

      From the app developer

      I have 2 node cluster of JIRA 7.9.0 (node1, node2). Each node is a docker. When I turn off node2 by stopping the docker container the other node still sees second node as active.

      I've checked that in clusternode table node2 is active. I've checked that in clusternodeheartbeat table beat data are not updated when node2 is down.

      The way I am getting node data:

      clusterManager.findLiveNodes();
      return clusterManager.getAllNodes();
      

      When instead of killing node2 I shut it down gracefully (/etc/init.d/jira stop), then node2 status in clusternode table is updated to OFFLINE.

      Jira should make use of clusternodehearbeat table to determine whether node is OFFLINE

      Tables data after node2 was terminated - not gracefully
      2.1) clusternode

      node_id,node_state,timestamp,ip,cache_listener_port,node_build_number,node_version
      node1,ACTIVE,1559562446911,30b2e7efd175,40001,79000,7.9.0
      node2,ACTIVE,1559834524044,bb7ae1186aa1,40001,79000,7.9.0
      

      2.2) clusternodeheartbeat

      node_id,heartbeat_time,database_time
      node1,1560462378461,1560462378462
      node2,1559852296782,1559852296782
      

      3) I killed a note to simulate a failure in DC cluster. In my case I stopped a docker on which Tomcat with Jira node was placed.

      Environment

      7.9.0

      Steps to Reproduce

      1. Have 2 nodes
      2. Terminate node 2 instead of shutting down gracefully

      Expected Results

      When node2 is down other nodes should see it as offline.

      Actual Results

      When node2 is down other nodes see it as online.

      Notes

      • I haven't personally replicated this. I'm moving this in this project for more expertise

      Workaround

      Currently there is no known workaround for this behavior. A workaround will be added here when available

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              acalantog Anne Calantog
              Votes:
              21 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated: