Uploaded image for project: 'HipChat'
  1. HipChat
  2. HCPUB-1083

Race condition in initial boot prevents login after HipChat Server setup

This issue belongs to an archived project. You can view it, but you can't modify it. Learn more

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Fixed
    • Icon: Medium Medium
    • None
    • None
    • Installation/Update
    • None

      Problem

      HipChat Server locked out admin from logging in successfully after a fresh server deployment

      Environment

      HipChat Server v2.0 build 1.4.2 (fresh OVA installs)

      Steps to reproduce

      1. Deploy HipChat Server and follow through the setup wizard
      2. You'll be prompted to restart the server
      3. Once server restarted, you'l be directed to the login page
      4. Specify your credentials that you have configured during setup wizard and login

      Expected Result

      User able to login to the server

      Actual Result

      User unable to login to the server

      The cause of the problem

      This issue appears to be a side effect of this bug: HipChat Server Chef run fails with 'Ran /etc/init.d/nginx start returned 1' Chef error. Because of that, there is a race condition in initial server boot up preventing user from logging in after HipChat Server setup. User is able to access/complete setup wizard however Chef (configuration manager) run failed and in affect to that Crowd service was unable to access the database successfully preventing authentication. This is seen in /var/log/hipchat/atlassian-crowd.log:

      2016-06-30 23:57:27,974 C3P0PooledConnectionPoolManager[identityToken->z8kflt9h19ffaq81ieite6|2fe4d2bd]-HelperThread-#2 WARN [mchange.v2.resourcepool.BasicResourcePool] com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask@637e128e -- Acquisition Attempt Failed!!! Clearing pending acquires. While trying to acquire a needed new resource, we failed to succeed more than the maximum number of allowed acquisition attempts (30). Last acquisition attempt exception: 
      com.mysql.jdbc.exceptions.jdbc4.MySQLNonTransientConnectionException: Could not create connection to database server. Attempted reconnect 3 times. Giving up.
      	at sun.reflect.GeneratedConstructorAccessor25.newInstance(Unknown Source)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
      	at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
      	at com.mysql.jdbc.Util.getInstance(Util.java:386)
      	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1015)
      	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:989)
      	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:975)
      	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:920)
      	at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2395)
      	at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2316)
      	at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
      	at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:47)
      	at sun.reflect.GeneratedConstructorAccessor23.newInstance(Unknown Source)
      	at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
      	at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
      	at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
      	at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:416)
      	at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:347)
      	at com.mchange.v2.c3p0.DriverManagerDataSource.getConnection(DriverManagerDataSource.java:146)
      	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:195)
      	at com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getPooledConnection(WrapperConnectionPoolDataSource.java:184)
      	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool$1PooledConnectionResourcePoolManager.acquireResource(C3P0PooledConnectionPool.java:200)
      	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquire(BasicResourcePool.java:1086)
      	at com.mchange.v2.resourcepool.BasicResourcePool.doAcquireAndDecrementPendingAcquiresWithinLockOnSuccess(BasicResourcePool.java:1073)
      	at com.mchange.v2.resourcepool.BasicResourcePool.access$800(BasicResourcePool.java:44)
      	at com.mchange.v2.resourcepool.BasicResourcePool$ScatteredAcquireTask.run(BasicResourcePool.java:1810)
      	at com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread.run(ThreadPoolAsynchronousRunner.java:648)
      Caused by: java.sql.SQLException: Access denied for user 'root'@'localhost' (using password: YES)
      	at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1086)
      	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4237)
      	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4169)
      	at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:928)
      	at com.mysql.jdbc.MysqlIO.proceedHandshakeWithPluggableAuthentication(MysqlIO.java:1750)
      	at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1290)
      	at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2493)
      	at com.mysql.jdbc.ConnectionImpl.connectWithRetries(ConnectionImpl.java:2334)
      	... 18 more
      

      Workaround

      Follow the suggested workaround mentioned in this bug report: HipChat Server Chef run fails with 'Ran /etc/init.d/nginx start returned 1' Chef error or apply the mentioned patch to fix the problem.

              Unassigned Unassigned
              hsuhailah Hanis Suhailah (Inactive)
              Archiver:
              mandreacchio Michael Andreacchio

                Created:
                Updated:
                Resolved:
                Archived: