Details
-
Bug
-
Resolution: Fixed
-
Low
-
HCS 2.2.2, HCS 2.2.6, HCS 2.2.9
-
None
-
Severity 2 - Major
Description
Summary
The tetra_sessions_cleanup.py script is failing to run due to unauthorized error:
2017-05-18T03:00:15.176797+00:00 chat ./tools/tetra_sessions_cleanup.py: [] ERROR: Error `unknown` calling internal URL 'http://<HCS_IP>:5231/disconnect?_internal=0'. Cause: [Failure instance: Traceback (failure with no frames): <class 'twisted.web.error.Error'>: 401 Unauthorized\n]
2017-05-18T03:00:15.176932+00:00 chat ./tools/tetra_sessions_cleanup.py: [] ERROR: tetra-proxy API request failed: 401 Unauthorized
This error prevents the script from running on a daily basis, which can cause 'rogue' sessions to pile up and ultimately prevent the user from logging in due to session limit (10 sessions). Once the sessions are cleared, the end user can log in again.
Environment
- HipChat Server 2.2.2
- HipChat Server 2.2.6
Steps to Reproduce
Unknown at this time
Expected Results
The script will run on a daily basis clearing out stuck, invalid and rogue user sessions.
Actual Results
The below exception is thrown in the /var/log/hipchat/tetra.log file:
2017-05-18T03:00:15.176797+00:00 chat ./tools/tetra_sessions_cleanup.py: [] ERROR: Error `unknown` calling internal URL 'http://<HCS_IP>:5231/disconnect?_internal=0'. Cause: [Failure instance: Traceback (failure with no frames): <class 'twisted.web.error.Error'>: 401 Unauthorized\n] 2017-05-18T03:00:15.176932+00:00 chat ./tools/tetra_sessions_cleanup.py: [] ERROR: tetra-proxy API request failed: 401 Unauthorized
As well as in the /var/log/hipchat/coral.log:
2017-05-19T20:03:22.717580+00:00 chat coral-0: [/user/18#REQC2hR03] ERROR: Error `unknown` calling internal URL 'http://<HCS_IP>:5221/disconnect?_internal=0'. Cause: [Failure instance: Traceback (failure with no frames): <class 'twisted.web.error.Error'>: 401 Unauthorized\n] 2017-05-19T20:03:22.717782+00:00 chat coral-0: [/user/18#REQC2hR03] ERROR: tetra-proxy API request failed: 401 Unauthorized
Notes
The ports in the logs point to a tetra service, which scale up the more vCPU's you add to the instance (up to 12). Original workarounds have been replaced with fixing the code inside the script manually. This needs to be only done once.
Workaround
The code identified inside of the tetra_sessions_cleanup.py can be replaced easily with the following one-liner. This will create a backup of the script first (stored at /hipchat-scm/fishtank/tools/tetra_sessions_cleanup.py.BAK) then apply the text change.
- Gain access to the HipChat Server (note that this bug affects versions 2.2.2 to 2.2.9)
- Run the following command:
sudo dont-blame-hipchat -c "cp /hipchat-scm/fishtank/tools/tetra_sessions_cleanup.py /hipchat-scm/fishtank/tools/tetra_sessions_cleanup.py.BAK && sed -i -e 's/p.get_memory_info/p.memory_info/g' /hipchat-scm/fishtank/tools/tetra_sessions_cleanup.py"
- Next, run the tetra_sessions_cleanup.py script manually:
cd /hipchat/tetra-app/current && env PYTHONPATH=/hipchat/tetra-app/current /hipchat/tetra-app/current/vendor/virtualenv/bin/python ./tools/tetra_sessions_cleanup.py 2>&1 >/dev/null
- Verify that no errors are present in the tetra.log:
tail -n400 /var/log/hipchat/tetra.log
How to tell if the tetra_sessions_cleanup.py script has already been updated?
If the tetra_sessions_cleanup.py script has already been updated, the following one-liner will return two lines:
sudo dont-blame-hipchat -c "grep -i "p.memory_info" /hipchat-scm/fishtank/tools/tetra_sessions_cleanup.py"
If the tetra_sessions_cleanup.py script has not been updated, the following one-liner will return two lines:
sudo dont-blame-hipchat -c "grep -i "p.get_memory_info" /hipchat-scm/fishtank/tools/tetra_sessions_cleanup.py"
If the script has yet to be updated, please run through the Workaround section above and test again.
Attachments
Issue Links
- relates to
-
HCPUB-3734 Hipchat Server: Cannot disconnect my stale chat sessions
- Closed