Uploaded image for project: 'HipChat'
  1. HipChat
  2. HCPUB-2767

HipChat Server slow to boot due to proxy environment

This issue belongs to an archived project. You can view it, but you can't modify it. Learn more

    XMLWordPrintable

    Details

    • Symptom Severity:
      Severity 2 - Major

      Description

      Summary

      In some environments where a proxy is present, HipChat Server will take an excessive amount of time to boot (~1-2hrs) depending on hardware. This delay is on every reboot.

      Environment

      HipChat Server v2.2.0

      Steps to Reproduce

      1. In a proxied environment, boot the HipChat Server VM
      2. In the boot output, notice that the boot process gets stuck for a considerable amount of time (in the example it's 45 minutes):
      /var/log/chef.log
      [Fri Jan 27 22:19:41 UTC 2017] SERVER role detected 
      PONG
      [Fri Jan 27 22:19:41 UTC 2017] Checking if the disk space has been increased and resizing accordingly... 
      [Fri Jan 27 23:05:49 UTC 2017] Starting consul 
      

      Can also get stuck here:

      /var/log/chef.log
      [2017-01-27T23:06:02+00:00] INFO: Syncing chef-solo and consul
      [2017-01-27T23:06:03+00:00] INFO: Completed syncing chef-solo and consul
      [2017-01-27T23:52:11+00:00] INFO: Creating consul session
      [2017-01-27T23:52:11+00:00] INFO: Attaining chef-lock
      

      Expected Results

      The HipChat Server boots in a timely manner (~5-10 minutes depending on hardware)

      Actual Results

      It takes a considerable amount of time to boot.

      Diagnostics

      This problem is rooted at /opt/atlassian/hipchat/sbin/_ovf_env.py which is used to check the HipChat Server's deployment type (Amazon EC2 or VMware/VirtualBox).

      The test below will indicate if a server is affected by this issue:

      Timed Test of _ovf_env.py
      1. Log into the HipChat Server command line.
      2. Run the follow command:
        time /usr/bin/sudo /opt/atlassian/hipchat/virtualenv/bin/python /opt/atlassian/hipchat/sbin/_ovf_env.py
        
      3. If the test reports back a total time of more than a few seconds, then it's possible you are affected by this issue.

      Workaround

      Bypass the VM check
      1. Boot the HipChat Server and wait until you have a command prompt
      2. Log into the HipChat Server command line
      3. Gain root:
        sudo dont-blame-hipchat
        
      4. Make a backup of _ovf_env.py
        cp /opt/atlassian/hipchat/sbin/_ovf_env.py /opt/atlassian/hipchat/sbin/_ovf_env.py.BAK
        
      5. Next, edit the original _ovf_env.py
        nano /opt/atlassian/hipchat/sbin/_ovf_env.py
        
      6. Locate the following code (starting with line 66).
        if __name__ == "__main__":
            platform = Platform()
            environmentals = None
            xml = fetch_ovf_metadata()
            if "HipChat" in xml:
                environmentals = extract_from_xml(xml)
            elif platform.test_ec2_environment():
                environmentals = platform.fetch_ec2_metadata()
        
            if environmentals:
                save_shell(environmentals)
        
      7. Once located, remove lines 67-76 (keeping line 66), then add pass at the end. Completed correctly, it'll look like:
        if __name__ == "__main__":
          pass
        
      8. Save the file (CTRL+X then 'Y' to save) and rerun the timed test again:
        time /usr/bin/sudo /opt/atlassian/hipchat/virtualenv/bin/python /opt/atlassian/hipchat/sbin/_ovf_env.py
        
      9. If successful, the test should pass in under a second. If the delay persists, please check steps 5, 6, and 7.
      10. If it passes the check, please reboot the VM and it should boot within a reasonable time.

        Attachments

          Activity

            People

            Assignee:
            Unassigned
            Reporter:
            dmaye@atlassian.com David Maye
            Archiver:
            mandreacchio Michael Andreacchio

              Dates

              Created:
              Updated:
              Resolved:
              Archived: