After upgrading HipChat Server, the clients will no longer connect and show the error "We timed out while loading HipChat", even though the web user interface works.
Upgrade to HCS 2.2.2
- Upgrade your instance, then reboot
- After the server is back online, users cannot connect using the desktop apps.
Users can connect after reboot
The desktop clients show an error "We timed out while loading HipChat"
One symptom of this bug can be seen via the following lines in /var/logs/hipchat/tetra.log
Instead, you're experiencing a flood of requests from end-users into the Server and are being rate limited (also called a reconnect storm). If possible, use your firewall to limit the connections into HipChat Server (on port 443) to help curb the rate limiting until the Tetra process is caught back up. Alternatively, you can also ask your users to log out of their clients (which stops the auto-connect mechanism). Do not restart Tetra or Punjab if you see this error as it will cause the storm to start up again.
Usually after ~10 minutes or so the Tetra processes catch up and start working normally.
Try running these workarounds one by one:
- Log into the HipChat Server command line
- Gain root:
- Set the python path
- Run the script:
- Once completed, attempt to log in using the desktop client. If you still cannot connect, follow the next workaround.
- Log into the web user interface
- If logged into the web chat app, click the avatar on the far right and click 'log out'. This will take you to the web user interface.
- Once here, select 'EDIT MY PROFILE', then on the left sidebar select 'CHAT SESSIONS'.
- Clear out any sessions that exist here.
- Attempt to log in using the desktop app. If you still cannot log in, please follow the next workaround.
- For Mac users, please clear out all your preferences according to the instructions at https://confluence.atlassian.com/hipchatkb/mac-desktop-app-clearing-preferences-751436224.html
- For Windows users, please follow these instructions to clear out the Window prefs:
- Quit the HipChat Client App
- Locate this directory using the Windows file explorer %USERPROFILE%\AppData\Local\Atlassian\HipChat
- Delete all folders and files from this directory, so you are left with an empty HipChat directory
- Locate this directory using the Windows file explorer %USERPROFILE%\AppData\Roaming\Atlassian
- Delete the HipChat.ini file
- Have the user re-connect to HipChat again. They will need to type in the URL for the server and re-enter their email and password to access the chat rooms.
- For Linux users:
- Quit the Linux client
- Wipe-out the cache:
- Remove the config file:
- Gain root access:
- Download this tar file to the server:
- Extract it:
- If possible, snapshot your server before executing the script
- Make the file executable, and run it:
- Verify that the patch took place by running a ps against punjab. The absence of "use_raw" at the end of the ps output will tell you that punjab is no longer running in raw mode: