Allow health check status to be set per-node

Problem Definition

The URL that is suggested for use in JIRA Data Center behind a load balancer, as mentioned in https://confluence.atlassian.com/display/ENTERPRISE/JIRA+Data+Center+Load+Balancer+examples states to use /status and it will return "state":"RUNNING" for a healthy node that should be in the pool. It is assumed in the documentation that customers have access to their load balancer to add/remove nodes from a pool. In cases where they do not have trivial access, the process for performing maintenance on 1 to n-1 nodes in a Data Center cluster can be expensive, both in terms of time and staff cost.

A suggested workaround has been to use a reverse proxy in front of each node such that the application team could drain connections by using a graceful shutdown of the proxy. This solution is reasonable but not always feasible, especially in cases when, for example, the use of Apache or other web servers is not permitted within the governance framework of a customer.

Suggested Solution

Add functionality to the cluster node section of the System info page (or even better, provide a cluster management interface!) to allow administrators to set a node to maintenance mode, effectively taking it out of the pool and draining connections, similar to how a foreground reindex behaves, but without the reindex. This will allow customers to perform tasks such as instance stop/start, thread/heap dumps, support investigations with a live instance, etc.