Details
High Description
Summary
After configurating Kerberos authentication to a SQL Server database, Bitbucket stops responding after a while and the application freezes.
Upgrading previous versions of Stash (that were running with Kerberos) to Bitbucket Server will cause this issue as well.
The reason why is that from Stash to Bitbucket, the software that handles the database pool has changed from BoneCP to HikariCP:
- Stash config properties
These properties control the database pool. The pool implementation used is BoneCP. Documentation for these settings can be found at: http://jolbox.com/configuration.html
- Bitbucket Server config properties
These properties control the database pool. The pool implementation used is HikariCP. Documentation for these settings can be found at: https://github.com/brettwooldridge/HikariCP/wiki/Configuration
To get a feel for how these settings really work in practice, the most relevant classes in HikariCP are:
com.zaxxer.hikari.HikariConfig Holds the configuration for the database pool and has documentation for the available settings.
com.zaxxer.hikari.pool.HikariPool Provides the database pool and manages connections.
com.zaxxer.hikari.util.ConnectionBag Holds references to open connections, whether in-use or idle.
Required fix:
We need to apply a fix in BBS that makes sure when HikariCP goes to open a connection it uses the webapp's ClassLoader, not the ClassLoader of whatever thread it's acquiring a connection on.
Environment
This problem occurs:
- when setting up Bitbucket Server (4.0+) with Kerberos running on Linux with MS-SQL Server as external database or;
- when setting up Bitbucket Server (4.0+) with any database configuration that requires access to a com.sun class and a plugin requesting a database connection requires the system to open a new one. I.e. BBS connected to an Oracle instance authenticating against an LDAP server.
- Kerberos and LDAP support are both implemented in com.sun classes, so they both trigger the issue
Only mixed-mode authentication is supported on Linux.
Integrated or Kerberos authentication to SQL Server is only supported on a Windows server.
The com.sun packages are not available to plugins, which results in the module not being available to ActiveObjects and causing the error above. On Windows, the Sun packages are not required as the DLL's in the JDBC driver provide everything needed.
Steps to Reproduce
Configure your environment with Kerberos as described on:
Expected Results
Application should work normally.
Actual Results
The following error can be seeing in the atlassian-bitbucket.log:
Caused by: javax.security.auth.login.LoginException: unable to find LoginModule class: com.sun.security.auth.module.Krb5LoginModule not found from bundle [com.atlassian.bitbucket.server.bitbucket-notification]
Or
Caused by: java.sql.SQLRecoverableException: IO Error: JNDI Package failure javax.naming.NoInitialContextException: Cannot instantiate class: com.sun.jndi.ldap.LdapCtxFactory [Root exception is java.lang.ClassNotFoundException: com.sun.jndi.ldap.LdapCtxFactory not found from bundle [com.atlassian.bitbucket.server.bitbucket-notification]]
Workaround
1- Add a property to <BITBUCKET_HOME>/shared/bitbucket.properties
Modify bitbucket.properties to add the following line:
db.pool.size.idle=${db.pool.size.max}
This will force the pool to open all of the possible connections immediately, which should prevent the unable to find LoginModule class error from happening in plugins.
Restart required
Note that after adding a property to bitbucket.properties an application restart is required.
2- Use mixed mode
If you'd like to stay on Linux, the solution is to allow mixed-mode authentication on the SQL Server and specify a username and password to connect to the database.
Alternatively, a migration to Windows will allow you to use the native DLL's.