Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-13379

Concurrent requests for deployments can result in exhausting DB connection pool


      Issue Summary

      When the number of concurrent deployment requests for a commit, are close to the maximum number of connections in the DB (db.pool.size.max), it can cause a lot of DB operations to fail due to unavailability of connections from the DB pool.

      This is reproducible on Data Center: yes

      Steps to Reproduce

      1. Create a project and a repository
      2. Push a commit in the repository
      3. Create deployments against that commit using different sequence numbers in about 70 different threads running in parallel. Each thread should try to create about 50 deployments in sequence i.e. it should make a POST request and wait for it's response before another request.
      4. Observe the failed HTTP requests made in Step 3 and the exceptions in the logs.

      Expected Results

      All the requests should eventually succeed as they have different sequence numbers for deployment creation request. Also the number of concurrent requests is lower than the DB pool size plus there is a timeout of 15 seconds for getting the DB connection.

      Actual Results

      Most of the requests fail after hanging for some time due to timeout while trying to get DB connection.

      The below exception is thrown many times in the atlassian-bitbucket.log file:

      Caused by: java.sql.SQLTransientConnectionException: bitbucket - Connection is not available, request timed out after 15000ms.
      	at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
      	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:197)
      	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:162)
      	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
      	at com.atlassian.stash.internal.hikari.ExtendedHikariDataSource.getConnection(ExtendedHikariDataSource.java:125)
      	at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.getTargetConnection(LazyConnectionDataSourceProxy.java:405)
      	at org.springframework.jdbc.datasource.LazyConnectionDataSourceProxy$LazyConnectionInvocationHandler.invoke(LazyConnectionDataSourceProxy.java:378)
      	at org.springframework.jdbc.datasource.TransactionAwareDataSourceProxy$TransactionAwareInvocationHandler.invoke(TransactionAwareDataSourceProxy.java:238)
      	at net.java.ao.DelegateConnectionHandler.delegate(DelegateConnectionHandler.java:118)
      	at net.java.ao.DelegateConnectionHandler.invoke(DelegateConnectionHandler.java:85)
      	at net.java.ao.DatabaseProvider.preparedStatement(DatabaseProvider.java:2306)
      	at net.java.ao.DatabaseProvider.executeInsertBatch(DatabaseProvider.java:1998)
      	at net.java.ao.DatabaseProvider.insertBatch(DatabaseProvider.java:1881)
      	at net.java.ao.EntityManager.create(EntityManager.java:456)
      	at com.atlassian.activeobjects.internal.EntityManagedActiveObjects.create(EntityManagedActiveObjects.java:109)
      	... 21 common frames omitted


      Increasing the DB connection pool size (db.pool.size.max) may help if the value is much higher than the number of concurrent deployment requests.

      Other notes
      Please note that the problem may be reproduced at even lesser number of concurrent deployment requests. The higher number is used here to help reproducing the issue consistently.

            esalter elisalter-atl
            mgoyal2@atlassian.com Manish
            0 Vote for this issue
            7 Start watching this issue