Details
High Description
Issue Summary
When the number of concurrent deployment requests for a commit, are close to the maximum number of connections in the DB (db.pool.size.max), it can cause a lot of DB operations to fail due to unavailability of connections from the DB pool.
This is reproducible on Data Center: yes
Steps to Reproduce
- Create a project and a repository
- Push a commit in the repository
- Create deployments against that commit using different sequence numbers in about 70 different threads running in parallel. Each thread should try to create about 50 deployments in sequence i.e. it should make a POST request and wait for it's response before another request.
- Observe the failed HTTP requests made in Step 3 and the exceptions in the logs.
Expected Results
All the requests should eventually succeed as they have different sequence numbers for deployment creation request. Also the number of concurrent requests is lower than the DB pool size plus there is a timeout of 15 seconds for getting the DB connection.
Actual Results
Most of the requests fail after hanging for some time due to timeout while trying to get DB connection.
The below exception is thrown many times in the atlassian-bitbucket.log file:
Caused by: java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms. at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696) at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197) at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162) at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100) at com.atlassian.stash.internal.hikari.ExtendedHikariDataSource.getConnection(ExtendedHikariDataSource.java:125) at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.getTargetConnection(LazyConnectionDataSourceProxy.java:405) at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:378) at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:238) at net.java.ao.DelegateConnectionHandler.delegate(DelegateConnectionHandler.java:118) at net.java.ao.DelegateConnectionHandler.invoke(DelegateConnectionHandler.java:85) at net.java.ao.DatabaseProvider.preparedStatement(DatabaseProvider.java:2306) at net.java.ao.DatabaseProvider.executeInsertBatch(DatabaseProvider.java:1998) at net.java.ao.DatabaseProvider.insertBatch(DatabaseProvider.java:1881) at net.java.ao.EntityManager.create(EntityManager.java:456) at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.create(EntityManagedActiveObjects.java:109) ... 21 common frames omitted
Workaround
Increasing the DB connection pool size (db.pool.size.max) may help if the value is much higher than the number of concurrent deployment requests.
Other notes
Please note that the problem may be reproduced at even lesser number of concurrent deployment requests. The higher number is used here to help reproducing the issue consistently.