Uploaded image for project: 'HipChat'
  1. HipChat
  2. HCPUB-2767

HipChat Server slow to boot due to proxy environment

This issue belongs to an archived project. You can view it, but you can't modify it. Learn more



    • Severity 2 - Major



      In some environments where a proxy is present, HipChat Server will take an excessive amount of time to boot (~1-2hrs) depending on hardware. This delay is on every reboot.


      HipChat Server v2.2.0

      Steps to Reproduce

      1. In a proxied environment, boot the HipChat Server VM
      2. In the boot output, notice that the boot process gets stuck for a considerable amount of time (in the example it's 45 minutes):
      [Fri Jan 27 22:19:41 UTC 2017] SERVER role detected 
      [Fri Jan 27 22:19:41 UTC 2017] Checking if the disk space has been increased and resizing accordingly... 
      [Fri Jan 27 23:05:49 UTC 2017] Starting consul 

      Can also get stuck here:

      [2017-01-27T23:06:02+00:00] INFO: Syncing chef-solo and consul
      [2017-01-27T23:06:03+00:00] INFO: Completed syncing chef-solo and consul
      [2017-01-27T23:52:11+00:00] INFO: Creating consul session
      [2017-01-27T23:52:11+00:00] INFO: Attaining chef-lock

      Expected Results

      The HipChat Server boots in a timely manner (~5-10 minutes depending on hardware)

      Actual Results

      It takes a considerable amount of time to boot.


      This problem is rooted at /opt/atlassian/hipchat/sbin/_ovf_env.py which is used to check the HipChat Server's deployment type (Amazon EC2 or VMware/VirtualBox).

      The test below will indicate if a server is affected by this issue:

      Timed Test of _ovf_env.py
      1. Log into the HipChat Server command line.
      2. Run the follow command:
        time /usr/bin/sudo /opt/atlassian/hipchat/virtualenv/bin/python /opt/atlassian/hipchat/sbin/_ovf_env.py
      3. If the test reports back a total time of more than a few seconds, then it's possible you are affected by this issue.


      Bypass the VM check
      1. Boot the HipChat Server and wait until you have a command prompt
      2. Log into the HipChat Server command line
      3. Gain root:
        sudo dont-blame-hipchat
      4. Make a backup of _ovf_env.py
        cp /opt/atlassian/hipchat/sbin/_ovf_env.py /opt/atlassian/hipchat/sbin/_ovf_env.py.BAK
      5. Next, edit the original _ovf_env.py
        nano /opt/atlassian/hipchat/sbin/_ovf_env.py
      6. Locate the following code (starting with line 66).
        if __name__ == "__main__":
            platform = Platform()
            environmentals = None
            xml = fetch_ovf_metadata()
            if "HipChat" in xml:
                environmentals = extract_from_xml(xml)
            elif platform.test_ec2_environment():
                environmentals = platform.fetch_ec2_metadata()
            if environmentals:
      7. Once located, remove lines 67-76 (keeping line 66), then add pass at the end. Completed correctly, it'll look like:
        if __name__ == "__main__":
      8. Save the file (CTRL+X then 'Y' to save) and rerun the timed test again:
        time /usr/bin/sudo /opt/atlassian/hipchat/virtualenv/bin/python /opt/atlassian/hipchat/sbin/_ovf_env.py
      9. If successful, the test should pass in under a second. If the delay persists, please check steps 5, 6, and 7.
      10. If it passes the check, please reboot the VM and it should boot within a reasonable time.




            Unassigned Unassigned
            dmaye@atlassian.com David Maye
            mandreacchio Michael Andreacchio