Gearman is not running and not monitored by Monit in Data Center

Summary

We have received reports of the following issues in Data Center: 

• Push and email notifications not being delivered to offline/away users. We have created a Knowledge Base article for this issue:
  Users are not receiving emails or push notifications on Data Center
• Room messages where offline/away users are not being delivered. We have a separate bug for this issue:
  HCPUB-3741 - Messages with mentions of offline or away users are not delivered
• API v1 and Bamboo notifications not working. Here's the separate bug we have for this issue:
  HCPUB-3757 - API v1 notifications and integrations do not work in Data Center

The issues above can happen if Gearman is down. We have observed that Gearman is either not brought up or dies silently in some new Data Center environments.

Environment

Hipchat Data Center 3.1.2 and older

Steps to Reproduce

We cannot reproduce the issue

Expected Results

Gearman should be up all the time as it's a key service to functioning of Hipchat.

Actual Results

• Gearman is down, the following ps command will return nothing:

  ps -ef | grep [/]usr/sbin/gearmand
  

• Gearman shows as not monitored by Monit in the output of:

  monit status | grep -A2 "Process 'gearmand'"
  

Workaround

Bring Gearman up as indicated in Users are not receiving emails or push notifications on Data Center