HipChat Data Center setup fails due to missing tables in the database

Problem

When performing a Manual Deployment of HipChat Data Center, the hipchat datacenter restart command fails to complete when run on any of the data center nodes.

The following is observed in /var/log/chef.log

Traceback (most recent call last):
  File "tools/update-database-schema", line 105, in <module>
    raise exception_raised
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) relation "cwd_application" does not exist
LINE 1: UPDATE cwd_application SET credential='{PKCS5S2}Ku0P4S+bXmT5...
               ^
 [SQL: "UPDATE cwd_application SET credential='{PKCS5S2}Ku0P4S+bXmT5iCqrauWbmP9jO3bcBIp2j4L/5DeWI0O6kKTz+FDtVK/mkuGa1h6x' WHERE cwd_application.id = '2'"]
---- End output of "bash"  "/tmp/chef-script20170712-74291-sbsi2g" ----
Ran "bash"  "/tmp/chef-script20170712-74291-sbsi2g" returned 1

Cause

The chef run failed because no cwd_application table existed in the hipchat database (when it should have).
The cwd_* tables are created via a script located on the nodes here: /opt/atlassian/hipchat/postgres/hipchat-crowd-schema.sh
This script is normally created, then executed, the first time hipchat datacenter restart is invoked.
If hipchat datacenter restart is executed on a node multiple times for whatever reason (i.e. due to other failures, etc), the hipchat-crowd-schema.sh script is never executed because it already exists on the filesystem.

Workaround

If the above error is found in the /var/log/chef.log, then the following steps should work around the issue:

• Log in to the command-line console of the affected node and execute:

  sudo dont-blame-hipchat
  
  rm -rf /opt/atlassian/hipchat/postgres/hipchat-crowd-schema.sh
  
  hipchat datacenter restart