Uploaded image for project: 'HipChat'
  1. HipChat
  2. HCPUB-3104

Tetra session cleanup script failing with 401 Unauthorized error

This issue belongs to an archived project. You can view it, but you can't modify it. Learn more

    XMLWordPrintable

    Details

    • Symptom Severity:
      Severity 2 - Major

      Description

      Summary

      The tetra_sessions_cleanup.py script is failing to run due to unauthorized error:

      2017-05-18T03:00:15.176797+00:00 chat ./tools/tetra_sessions_cleanup.py: [] ERROR: Error `unknown` calling internal URL 'http://<HCS_IP>:5231/disconnect?_internal=0'. Cause: [Failure instance: Traceback (failure with no frames): <class 'twisted.web.error.Error'>: 401 Unauthorized\n]
      2017-05-18T03:00:15.176932+00:00 chat ./tools/tetra_sessions_cleanup.py: [] ERROR: tetra-proxy API request failed: 401 Unauthorized
      

      This error prevents the script from running on a daily basis, which can cause 'rogue' sessions to pile up and ultimately prevent the user from logging in due to session limit (10 sessions). Once the sessions are cleared, the end user can log in again.

      Environment

      • HipChat Server 2.2.2
      • HipChat Server 2.2.6

      Steps to Reproduce

      Unknown at this time

      Expected Results

      The script will run on a daily basis clearing out stuck, invalid and rogue user sessions.

      Actual Results

      The below exception is thrown in the /var/log/hipchat/tetra.log file:

      2017-05-18T03:00:15.176797+00:00 chat ./tools/tetra_sessions_cleanup.py: [] ERROR: Error `unknown` calling internal URL 'http://<HCS_IP>:5231/disconnect?_internal=0'. Cause: [Failure instance: Traceback (failure with no frames): <class 'twisted.web.error.Error'>: 401 Unauthorized\n]
      2017-05-18T03:00:15.176932+00:00 chat ./tools/tetra_sessions_cleanup.py: [] ERROR: tetra-proxy API request failed: 401 Unauthorized
      

      As well as in the /var/log/hipchat/coral.log:

      2017-05-19T20:03:22.717580+00:00 chat coral-0: [/user/18#REQC2hR03] ERROR: Error `unknown` calling internal URL 'http://<HCS_IP>:5221/disconnect?_internal=0'. Cause: [Failure instance: Traceback (failure with no frames): <class 'twisted.web.error.Error'>: 401 Unauthorized\n]
      2017-05-19T20:03:22.717782+00:00 chat coral-0: [/user/18#REQC2hR03] ERROR: tetra-proxy API request failed: 401 Unauthorized
      

      Notes

      The ports in the logs point to a tetra service, which scale up the more vCPU's you add to the instance (up to 12). Original workarounds have been replaced with fixing the code inside the script manually. This needs to be only done once.

      Workaround

      The code identified inside of the tetra_sessions_cleanup.py can be replaced easily with the following one-liner. This will create a backup of the script first (stored at /hipchat-scm/fishtank/tools/tetra_sessions_cleanup.py.BAK) then apply the text change.

      1. Gain access to the HipChat Server (note that this bug affects versions 2.2.2 to 2.2.9)
      2. Run the following command:
        sudo dont-blame-hipchat -c "cp /hipchat-scm/fishtank/tools/tetra_sessions_cleanup.py /hipchat-scm/fishtank/tools/tetra_sessions_cleanup.py.BAK && sed -i -e 's/p.get_memory_info/p.memory_info/g' /hipchat-scm/fishtank/tools/tetra_sessions_cleanup.py"
        
      3. Next, run the tetra_sessions_cleanup.py script manually:
        cd /hipchat/tetra-app/current &&     env PYTHONPATH=/hipchat/tetra-app/current       /hipchat/tetra-app/current/vendor/virtualenv/bin/python ./tools/tetra_sessions_cleanup.py 2>&1 >/dev/null
        
      4. Verify that no errors are present in the tetra.log:
        tail -n400 /var/log/hipchat/tetra.log 
        

      How to tell if the tetra_sessions_cleanup.py script has already been updated?

      If the tetra_sessions_cleanup.py script has already been updated, the following one-liner will return two lines:

      sudo dont-blame-hipchat -c "grep -i "p.memory_info" /hipchat-scm/fishtank/tools/tetra_sessions_cleanup.py"
      

      If the tetra_sessions_cleanup.py script has not been updated, the following one-liner will return two lines:

      sudo dont-blame-hipchat -c "grep -i "p.get_memory_info" /hipchat-scm/fishtank/tools/tetra_sessions_cleanup.py"
      

      If the script has yet to be updated, please run through the Workaround section above and test again.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              dmaye@atlassian.com David Maye
              Archiver:
              mandreacchio Michael Andreacchio

                Dates

                Created:
                Updated:
                Resolved:
                Archived: