Add REST APIs methods to clean old node ids in JIRA Data Center

Problem

Jira DC administrators experience a problem with tons of records about their cluster nodes on System Info page in the admin panel. DC enterprise customers restart or kill or add new nodes as part of the everyday activities connected with managing their cluster. All those activities lead to the creation of records in the database and on System Info page under Cluster Nodes bookmark. As a result, administrators see not the actual state of their clusters, but all active & nonactive nodes in a long list that is hard for them to understand and brings more confusion than visibility. This problem becomes more significant in the case of AWS deployments.

Suggested solution

Since Jira 8.1.0 we would like to provide the experimental REST API which allows Admins in Jira Data Center to manage the state of the cluster. We would like to provide three methods:

• GET /rest/api/2/cluster/nodes - return all nodes in cluster in JSON format.
• PUT  /rest/api/2/cluster/node/{nodeId}/offline - change the node's state to OFFLINE if the node is reporting as ACTIVE, but is non "alive" (alive = false). 
  • This step could have a negative effect on the cluster consistency, so it's under the discretion of Jira admin. Make sure the node will not rejoin the cluster, in another case it will have non-consistent index/cache data. The node can be restarted.
  • The node is marked as non "alive" after missing the heartbeat timeout (default 5min).
  • Thus the node potentially could be marked as OFFLINE only after heartbeat timeout.
  •  Remember this function is not able to stop Jira instance.
• DELETE /rest/api/2/cluster/node/{nodeId} - delete the node from the cluster and database, if state of node is OFFLINE.
  • Safe to execute

Only Jira System Administrators can execute those methods.

Example

Example response of GET method:

[
 {
  "nodeId": "node3",
  "state": "ACTIVE",
  "lastStateChangeTimestamp": 1551080754927,
  "ip": "localhost",
  "cacheListenerPort": 40001,
  "nodeBuildNumber": 801000,
  "nodeVersion": "8.1.0-SNAPSHOT",
  "alive": true
 },{
  "nodeId": "node1",
  "state": "ACTIVE",
  "lastStateChangeTimestamp": 1551080840469,
  "ip": "localhost",
  "cacheListenerPort": 40001,
  "nodeBuildNumber": 801000,
  "nodeVersion": "8.1.0-SNAPSHOT",
  "alive": false
 }
]

Example usage:

1. Make a request - GET /rest/api/2/cluster/nodes
2. Jira returns a list of nodes*
3. Find "ACTIVE" and "alive"=false nodes ID in the previous response - e.g. "node1"
4. Make sure that nodes (e.g. "node1") are not running in your environment and will not back to the cluster in the future.
   • They should be stoped
5. Make a request - PUT /rest/api/2/cluster/node/node1/offline
6. Make a request - GET /rest/api/2/cluster/nodes
7. Find out that "node1" now has "OFFLINE" state
8. Make a request -  DELETE /rest/api/2/cluster/node/node1
9. Make a request - GET /rest/api/2/cluster/nodes
10. Find out that "node1" was deleted