Details
-
Bug
-
Resolution: Fixed
-
Medium
-
None
-
HCS 1.4.1, HCS 1.4.2
-
None
Description
Our recommended resolution for this issue is to upgrade to HipChat Server v1.4.3, which is now available on the Production Channel. Please see instructions for upgrading: https://confluence.atlassian.com/hc/upgrading-hipchat-server-606306347.html
Summary
After a fresh installation or upgrade to newer version, the terminal//var/log/chef.log displays the following error:
[2016-06-27T16:38:37+02:00] INFO: bash[monit_monitor_mysqld] ran successfully [2016-06-27T16:38:37+02:00] ERROR: Running exception handlers [2016-06-27T16:38:37+02:00] ERROR: Exception handlers complete [2016-06-27T16:38:37+02:00] FATAL: Stacktrace dumped to /usr/local/src/chef-stacktrace.out [2016-06-27T16:38:37+02:00] ERROR: service[nginx] (nginx::package line 91) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1' ---- Begin output of /etc/init.d/nginx start ---- STDOUT: STDERR: ---- End output of /etc/init.d/nginx start ---- Ran /etc/init.d/nginx start returned 1 [2016-06-27T16:38:37+02:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
Symptoms
- the web chat application failing to load with the 'something went wrong' error
- desktop client connectivity problems
- email notifications failing
- push notifications failing
- login failures after a fresh installation or upgrade
Environment
Fresh installations of/upgrades to:
- HipChat Server v1.4.0
- HipChat Server v1.4.1
- HipChat Server v1.4.2
Steps to Reproduce
Does not always reproduce
- Install/upgrade HipChat Server with an affected version
- Chef run fails to complete due to nginx error
Expected Results
Chef completes and the upgrade/install is successful
Actual Results
The below exception is thrown in the /var/log/chef.log file:
[2016-06-27T16:38:37+02:00] INFO: bash[monit_monitor_mysqld] ran successfully [2016-06-27T16:38:37+02:00] ERROR: Running exception handlers [2016-06-27T16:38:37+02:00] ERROR: Exception handlers complete [2016-06-27T16:38:37+02:00] FATAL: Stacktrace dumped to /usr/local/src/chef-stacktrace.out [2016-06-27T16:38:37+02:00] ERROR: service[nginx] (nginx::package line 91) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1' ---- Begin output of /etc/init.d/nginx start ---- STDOUT: STDERR: ---- End output of /etc/init.d/nginx start ---- Ran /etc/init.d/nginx start returned 1 [2016-06-27T16:38:37+02:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)
Workaround
These workarounds will cause a brief service interruption (5-10 minutes) while services inside of HipChat Server are reconfigured. We recommend setting a maintenance window and backing up HipChat Server beforehand.
Permanent patch
Log into the HipChat Server terminal/command-line interface and run the following commands:
sudo dont-blame-hipchat
cd /home/admin
wget https://s3.amazonaws.com/hipchat-server-stable/utils/hcpub-1045.tar.gz
tar xf hcpub-1045.tar.gz
cd hcpub-1045
./hcpub-1045-nginx-fix.sh
Temporary resolution
Log into the HipChat Server terminal/command-line interface and run the following commands:
sudo dont-blame-hipchat cs
If the nginx errors are still occurring in the /var/log/chef.log file, please open a new support ticket and include a diagnostic bundle/password from the HipChat Server for further investigation by running the following command at the HipChat Server terminal/command-line interface:
hipchat diagnostics -b