Summary

The tetra_sessions_cleanup.py script is failing to run due to unauthorized error:

2017-05-18T03:00:15.176797+00:00 chat ./tools/tetra_sessions_cleanup.py: [] ERROR: Error `unknown` calling internal URL 'http://<HCS_IP>:5231/disconnect?_internal=0'. Cause: [Failure instance: Traceback (failure with no frames): <class 'twisted.web.error.Error'>: 401 Unauthorized\n]
2017-05-18T03:00:15.176932+00:00 chat ./tools/tetra_sessions_cleanup.py: [] ERROR: tetra-proxy API request failed: 401 Unauthorized

This error prevents the script from running on a daily basis, which can cause 'rogue' sessions to pile up and ultimately prevent the user from logging in due to session limit (10 sessions). Once the sessions are cleared, the end user can log in again.

Environment

• HipChat Server 2.2.2
• HipChat Server 2.2.6

Steps to Reproduce

Unknown at this time

Expected Results

The script will run on a daily basis clearing out stuck, invalid and rogue user sessions.

Actual Results

The below exception is thrown in the /var/log/hipchat/tetra.log file:

2017-05-18T03:00:15.176797+00:00 chat ./tools/tetra_sessions_cleanup.py: [] ERROR: Error `unknown` calling internal URL 'http://<HCS_IP>:5231/disconnect?_internal=0'. Cause: [Failure instance: Traceback (failure with no frames): <class 'twisted.web.error.Error'>: 401 Unauthorized\n]
2017-05-18T03:00:15.176932+00:00 chat ./tools/tetra_sessions_cleanup.py: [] ERROR: tetra-proxy API request failed: 401 Unauthorized

As well as in the /var/log/hipchat/coral.log:

2017-05-19T20:03:22.717580+00:00 chat coral-0: [/user/18#REQC2hR03] ERROR: Error `unknown` calling internal URL 'http://<HCS_IP>:5221/disconnect?_internal=0'. Cause: [Failure instance: Traceback (failure with no frames): <class 'twisted.web.error.Error'>: 401 Unauthorized\n]
2017-05-19T20:03:22.717782+00:00 chat coral-0: [/user/18#REQC2hR03] ERROR: tetra-proxy API request failed: 401 Unauthorized

Notes

The ports in the logs point to a tetra service, which scale up the more vCPU's you add to the instance (up to 12). Original workarounds have been replaced with fixing the code inside the script manually. This needs to be only done once.

Workaround

The code identified inside of the tetra_sessions_cleanup.py can be replaced easily with the following one-liner. This will create a backup of the script first (stored at /hipchat-scm/fishtank/tools/tetra_sessions_cleanup.py.BAK) then apply the text change.

1. Gain access to the HipChat Server (note that this bug affects versions 2.2.2 to 2.2.9)
2. Run the following command:

   sudo dont-blame-hipchat -c "cp /hipchat-scm/fishtank/tools/tetra_sessions_cleanup.py /hipchat-scm/fishtank/tools/tetra_sessions_cleanup.py.BAK && sed -i -e 's/p.get_memory_info/p.memory_info/g' /hipchat-scm/fishtank/tools/tetra_sessions_cleanup.py"
   

3. Next, run the tetra_sessions_cleanup.py script manually:

   cd /hipchat/tetra-app/current &&     env PYTHONPATH=/hipchat/tetra-app/current       /hipchat/tetra-app/current/vendor/virtualenv/bin/python ./tools/tetra_sessions_cleanup.py 2>&1 >/dev/null
   

4. Verify that no errors are present in the tetra.log:

   tail -n400 /var/log/hipchat/tetra.log 
   

How to tell if the tetra_sessions_cleanup.py script has already been updated?

If the tetra_sessions_cleanup.py script has already been updated, the following one-liner will return two lines:

sudo dont-blame-hipchat -c "grep -i "p.memory_info" /hipchat-scm/fishtank/tools/tetra_sessions_cleanup.py"

If the tetra_sessions_cleanup.py script has not been updated, the following one-liner will return two lines:

sudo dont-blame-hipchat -c "grep -i "p.get_memory_info" /hipchat-scm/fishtank/tools/tetra_sessions_cleanup.py"

If the script has yet to be updated, please run through the Workaround section above and test again.