After upgrading, HipChat Server web pages fail to load with a s3.php:5 error

Problem

After upgrading the HipChat Server, every page fails to load from the web browser with the following errors:

PHP Error: Undefined index: use_filesystem on /hipchat-scm/web/application/config/s3.php:5

0: /hipchat-scm/web/application/config/s3.php:5 -> exception_handler()
1: /hipchat-scm/web/system/core/Kohana.php:454 -> require()
2: /hipchat-scm/web/system/core/Kohana.php:358 -> config_load()
3: /hipchat-scm/web/application/models/group.php:1308 -> config()
4: /hipchat-scm/web/application/models/group.php:1292 -> get_absolute_avatar_url()
5: /hipchat-scm/web/application/views/_header.php:28 -> getAvatarUrl()
6: /hipchat-scm/web/application/views/layout_focused.php:4 -> include()
7: /hipchat-scm/web/system/libraries/Controller.php:84 -> include()
8: /hipchat-scm/web/system/libraries/View.php:266 -> _kohana_load_view()
9: /hipchat-scm/web/system/controllers/template.php:50 -> render()
10: /hipchat-scm/web/application/controllers/application.php:366 -> _render()
11: in Application_Controller::_render()
12: /hipchat-scm/web/system/core/Event.php:209 -> call_user_func()
13: /hipchat-scm/web/system/core/Kohana.php:289 -> run()
14: in Kohana::instance()
15: /hipchat-scm/web/system/core/Event.php:209 -> call_user_func()
16: /hipchat-scm/web/system/core/Bootstrap.php:55 -> run()
17: /hipchat-scm/web/www/index.php:109 -> require()

Environment

• HipChat Servers running version 2.2.1 in an Amazon AWS environment.
• Server was upgraded to 2.2.2 or later. The upgrade itself went without issue.

Root Cause

• An incorrect AMI image was published briefly to Amazon. This resulted in a HipChat Server deployment running unstable versions of HipChat Server code.

Workaround

• If the problem is encountered after an upgrade from 2.2.1, please follow the below steps:

1. Run "dpkg l | grep hc" from the server's command line.

   dpkg -l | grep hc-
   ii  hc-chef-repo                           2017.03.09.232643-fba29e7           all          config management for HipChat
   ii  hc-client-bundle                       2016.11.09.162543-366c47e           all          HipChat desktop and mobile clients
   ii  hc-consul                              2016.11.15.023757-45df12a           all          Consul for HipChat
   ii  hc-crowd                               2016.11.01.234254-19fb239           all          Atlassian Crowd for HipChat
   ii  hc-cumulus                             2017.01.11.141326-edb3c4d           all          S3 functionality for HipChat
   ii  hc-curler                              2017.01.10.214955-fb007fd           all          HipChat Curler
   ii  hc-emoticons                           2016.11.23.020131-caa9654           all          HipChat Emoticons
   ii  hc-fishtank                            2017.02.16.150859-fea0731           amd64        HipChat Services
   ii  hc-punjab                              2017.02.15.234325-de0a6cd           amd64        BOSH server
   ii  hc-s3-signer                           2016.11.22.143244-02f772b           all          S3 Read request signer for HipChat
   ii  hc-scissortail                         2017.02.15.234340-481af3b           all          HipChat data migration
   ii  hc-server-cli                          2017.02.22.190221-ca38b3c           amd64        HipChat Server CLI
   ii  hc-web                                 2017.02.21.130048-0fa73a8           all          web-based functionality for HipChat
   ii  hc-webcore                             2017.02.20.034200-c717112           all          javascript client for HipChat
   

2. If the hc-fishtank hc-punjab hc-scissortail hc-server-cli hc-web, and hc-webcore packages match the versions listed above, then you are affected by this issue.
3. Run the following commands to confirm that the correct hipchat server packages are present on the server's on-disk repository. The correct package versions are listed in the output below:

   ls -ahl /file_store/archive/pool/main/h/hc-web
   rw-rr- 1 root root 43M Mar 9 23:31 hc-web_2017.02.08.202236-2ca9673_all.deb
   
   ls -ahl /file_store/archive/pool/main/h/hc-fishtank
   total 22M
   drwxr-xr-x  2 root root 4.0K Mar  9 23:31 .
   drwxr-xr-x 23 root root 4.0K Mar  9 23:31 ..
   -rw-r--r--  1 root root  22M Mar  9 23:31 hc-fishtank_2017.02.08.001337-b1f002b_all.de
   
    ls -ahl /file_store/archive/pool/main/h/hc-punjab
   total 8.0M
   drwxr-xr-x  2 root root 4.0K Mar  9 23:31 .
   drwxr-xr-x 23 root root 4.0K Mar  9 23:31 ..
   -rw-r--r--  1 root root 8.0M Mar  9 23:31 hc-punjab_2017.01.11.193404-b73e0b1_all.deb
   
    ls -ahl /file_store/archive/pool/main/h/hc-scissortail
   total 7.4M
   drwxr-xr-x  2 root root 4.0K Mar  9 23:31 .
   drwxr-xr-x 23 root root 4.0K Mar  9 23:31 ..
   -rw-r--r--  1 root root 7.4M Mar  9 23:31 hc-scissortail_2017.01.27.162526-6c8bda7_all.deb
   
   ls -ahl /file_store/archive/pool/main/h/hc-server-cli/
   total 6.7M
   drwxr-xr-x  2 root root 4.0K Mar  9 23:31 .
   drwxr-xr-x 23 root root 4.0K Mar  9 23:31 ..
   -rw-r--r--  1 root root 6.7M Mar  9 23:31 hc-server-cli_2017.02.14.194429-c6b859c_amd64.deb
   
   ls -ahl /file_store/archive/pool/main/h/hc-webcore/
   total 5.4M
   drwxr-xr-x  2 root root 4.0K Mar  9 23:31 .
   drwxr-xr-x 23 root root 4.0K Mar  9 23:31 ..
   -rw-r--r--  1 root root 5.4M Mar  9 23:31 hc-webcore_2017.01.31.212121-3b14ecf_all.deb
   

4. Snapshot your instance before proceeding with the next steps
5. gain root: sudo dont-blame-hipchat
6. Remove the incorrect packages using this command:

   apt-get remove hc-fishtank hc-punjab hc-scissortail hc-server-cli hc-web hc-webcore
   

7. Type "y" when prompted. You'll see a bunch of warnings when the packages are uninstalled. It's fine to ignore them.
8. Reinstall the correct hc-server-cli package by running:

   dpkg -i /file_store/archive/pool/main/h/hc-server-cli/hc-server-cli_2017.02.14.194429-c6b859c_amd64.deb
   

9. Once the hc-server-cli package is reinstalled, please run this command, which will kick off a chef run, install the corect packages and configurations, and restart all services:

   hipchat upgrade --restart
   

10. From there, please log in and check to see if normal operations on the server have resumed.