Problem

If the IP address or hostname changes for the external Postgres host, the changes do not seem to propagate to the crowd.cfg.xml configuration.

This results in crowd not starting properly and users cannot log into HipChat Data Center.

Steps to Reproduce

1) Update "servers" section of /hipchat/config/site.json to point to the new address of the Postgres database, for example:

{
    "environment": "btf",
    "hostname": "hipchat",
    "ipaddress": "192.168.122.4",
    "databases": {
        "hipchat_postgres": {
            "user": "hipchat",
            "pass": "hipchat",
            "schema": "hipchat",
            "servers": [
                "172.22.48.113:5432"
            ]
        }
    },
    "server_id": "ca5277e3-9c37-4f77-af5b-4cbf0089e941"
}

2) If the same system that hosts postgres, also houses Redis and NFS, also ensure that the configurations table in the HipChat Postgres database is updated to point to the new addresses for the respective services.
3) Run hipchat datacenter restart
4) Attempt to log in to HipChat.

Actual Results

• Users cannot log in to HipChat Data Center since Crowd is not starting properly.
• Their username and password may appear to be accepted, but, users are presented with a "Staring Services..." screen and, shortly after, are sent back to the login page.
• The /var/log/hipchat/atlassian-crowd.log may show the following error, complaining of the inability to connect to the postgres database:

  2018-05-09 18:43:49,465 C3P0PooledConnectionPoolManager[identityToken->i19lvlrjmeki10pzpke|130cb68c]-HelperThread-#2 WARN [mchange.v2.resourcepool.BasicResourcePool] com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@62b4cf5e -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: 
  org.postgresql.util.PSQLException: The connection attempt failed.
  	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:272)
  	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:52)
  	at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:216)
  ....
  

Expected Results

• Embedded Crowd should not crash and users should be able to log in.

Root Cause

• The crowd.cfg.xml file was not updated to reflect the address change for the postgres database.

Workaround

• Log in to the hipchat data center node command line and gain root by running sudo dont-blame-hipchat
• Edit the /file_store/shared/crowd/shared/crowd.cfg.xml and update the following line with the correct hostname or IP address of the postgres server:

  <property name="hibernate.connection.url">jdbc:postgresql://172.16.53.158:5432/hipchat_postgres</property>
  

• Also, on each Data Center node, check the /etc/crowd/crowd.cfg.xml file. If the database address is also incorrect, please fix it there as well.
• Restart Crowd by running the following on each data center node

  /bin/bash /opt/atlassian/hipchat/sbin/crowd_restart.sh
  

• Attempt to log in to HipChat.