Uploaded image for project: 'HipChat'
  1. HipChat
  2. HCPUB-1045

HipChat Server Chef run fails with 'Ran /etc/init.d/nginx start returned 1' Chef error

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: Medium
    • Resolution: Fixed
    • Affects Version/s: HCS 1.4.1, HCS 1.4.2
    • Fix Version/s: None
    • Component/s: Installation/Update
    • Labels:
      None
    • Last commented by user?:
      true
    • Platform:
      HipChat Server

      Description

      Update as of July 14th, 2016

      Our recommended resolution for this issue is to upgrade to HipChat Server v1.4.3, which is now available on the Production Channel. Please see instructions for upgrading: https://confluence.atlassian.com/hc/upgrading-hipchat-server-606306347.html

      Summary

      After a fresh installation or upgrade to newer version, the terminal//var/log/chef.log displays the following error:

      [2016-06-27T16:38:37+02:00] INFO: bash[monit_monitor_mysqld] ran successfully
      [2016-06-27T16:38:37+02:00] ERROR: Running exception handlers
      [2016-06-27T16:38:37+02:00] ERROR: Exception handlers complete
      [2016-06-27T16:38:37+02:00] FATAL: Stacktrace dumped to /usr/local/src/chef-stacktrace.out
      [2016-06-27T16:38:37+02:00] ERROR: service[nginx] (nginx::package line 91) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
      ---- Begin output of /etc/init.d/nginx start ----
      STDOUT: 
      STDERR: 
      ---- End output of /etc/init.d/nginx start ----
      Ran /etc/init.d/nginx start returned 1
      [2016-06-27T16:38:37+02:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

      Symptoms

      • the web chat application failing to load with the 'something went wrong' error
      • desktop client connectivity problems
      • email notifications failing
      • push notifications failing
      • login failures after a fresh installation or upgrade

      Environment

      Fresh installations of/upgrades to:

      • HipChat Server v1.4.0
      • HipChat Server v1.4.1
      • HipChat Server v1.4.2

      Steps to Reproduce

      Does not always reproduce

      1. Install/upgrade HipChat Server with an affected version
      2. Chef run fails to complete due to nginx error

      Expected Results

      Chef completes and the upgrade/install is successful

      Actual Results

      The below exception is thrown in the /var/log/chef.log file:

      [2016-06-27T16:38:37+02:00] INFO: bash[monit_monitor_mysqld] ran successfully
      [2016-06-27T16:38:37+02:00] ERROR: Running exception handlers
      [2016-06-27T16:38:37+02:00] ERROR: Exception handlers complete
      [2016-06-27T16:38:37+02:00] FATAL: Stacktrace dumped to /usr/local/src/chef-stacktrace.out
      [2016-06-27T16:38:37+02:00] ERROR: service[nginx] (nginx::package line 91) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
      ---- Begin output of /etc/init.d/nginx start ----
      STDOUT: 
      STDERR: 
      ---- End output of /etc/init.d/nginx start ----
      Ran /etc/init.d/nginx start returned 1
      [2016-06-27T16:38:37+02:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

      Workaround

      These workarounds will cause a brief service interruption (5-10 minutes) while services inside of HipChat Server are reconfigured. We recommend setting a maintenance window and backing up HipChat Server beforehand.

      Permanent patch

      Log into the HipChat Server terminal/command-line interface and run the following commands:

      sudo dont-blame-hipchat
      cd /home/admin
      wget https://s3.amazonaws.com/hipchat-server-stable/utils/hcpub-1045.tar.gz
      tar xf hcpub-1045.tar.gz
      cd hcpub-1045
      ./hcpub-1045-nginx-fix.sh
      Temporary resolution

      Log into the HipChat Server terminal/command-line interface and run the following commands:

      sudo dont-blame-hipchat
      cs

      If the nginx errors are still occurring in the /var/log/chef.log file, please open a new support ticket and include a diagnostic bundle/password from the HipChat Server for further investigation by running the following command at the HipChat Server terminal/command-line interface:

      hipchat diagnostics -b

        Attachments

          Issue Links

            Activity

              People

              • Votes:
                4 Vote for this issue
                Watchers:
                18 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:
                  Last commented:
                  36 weeks, 5 days ago