HipChat Server slow to boot due to proxy environment

Summary

In some environments where a proxy is present, HipChat Server will take an excessive amount of time to boot (~1-2hrs) depending on hardware. This delay is on every reboot.

Environment

HipChat Server v2.2.0

Steps to Reproduce

1. In a proxied environment, boot the HipChat Server VM
2. In the boot output, notice that the boot process gets stuck for a considerable amount of time (in the example it's 45 minutes):

[Fri Jan 27 22:19:41 UTC 2017] SERVER role detected 
PONG
[Fri Jan 27 22:19:41 UTC 2017] Checking if the disk space has been increased and resizing accordingly... 
[Fri Jan 27 23:05:49 UTC 2017] Starting consul 

Can also get stuck here:

[2017-01-27T23:06:02+00:00] INFO: Syncing chef-solo and consul
[2017-01-27T23:06:03+00:00] INFO: Completed syncing chef-solo and consul
[2017-01-27T23:52:11+00:00] INFO: Creating consul session
[2017-01-27T23:52:11+00:00] INFO: Attaining chef-lock

Expected Results

The HipChat Server boots in a timely manner (~5-10 minutes depending on hardware)

Actual Results

It takes a considerable amount of time to boot.

Diagnostics

This problem is rooted at /opt/atlassian/hipchat/sbin/_ovf_env.py which is used to check the HipChat Server's deployment type (Amazon EC2 or VMware/VirtualBox).

The test below will indicate if a server is affected by this issue:

Timed Test of _ovf_env.py

1. Log into the HipChat Server command line.
2. Run the follow command:

   time /usr/bin/sudo /opt/atlassian/hipchat/virtualenv/bin/python /opt/atlassian/hipchat/sbin/_ovf_env.py
   

3. If the test reports back a total time of more than a few seconds, then it's possible you are affected by this issue.

Workaround

Bypass the VM check

1. Boot the HipChat Server and wait until you have a command prompt
2. Log into the HipChat Server command line
3. Gain root:

   sudo dont-blame-hipchat
   

4. Make a backup of _ovf_env.py

   cp /opt/atlassian/hipchat/sbin/_ovf_env.py /opt/atlassian/hipchat/sbin/_ovf_env.py.BAK
   

5. Next, edit the original _ovf_env.py

   nano /opt/atlassian/hipchat/sbin/_ovf_env.py
   

6. Locate the following code (starting with line 66).

   if __name__ == "__main__":
       platform = Platform()
       environmentals = None
       xml = fetch_ovf_metadata()
       if "HipChat" in xml:
           environmentals = extract_from_xml(xml)
       elif platform.test_ec2_environment():
           environmentals = platform.fetch_ec2_metadata()
   
       if environmentals:
           save_shell(environmentals)
   

7. Once located, remove lines 67-76 (keeping line 66), then add pass at the end. Completed correctly, it'll look like:

   if __name__ == "__main__":
     pass
   

8. Save the file (CTRL+X then 'Y' to save) and rerun the timed test again:

   time /usr/bin/sudo /opt/atlassian/hipchat/virtualenv/bin/python /opt/atlassian/hipchat/sbin/_ovf_env.py
   

9. If successful, the test should pass in under a second. If the delay persists, please check steps 5, 6, and 7.
10. If it passes the check, please reboot the VM and it should boot within a reasonable time.