Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-19838

Add improvements to cluster-wide locking of pull requests to avoid database connections exhaustion

XMLWordPrintable

    • Icon: Suggestion Suggestion
    • Resolution: Unresolved
    • None
    • Pull Requests
    • None
    • 6
    • We collect Bitbucket feedback from various sources, and we evaluate what we've collected when planning our product roadmap. To understand how this piece of feedback will be reviewed, see our Implementation of New Features Policy.

      Context

      It is possible that clustering issues (e.g. split-brain scenario) may lead to threads calling methods on cluster-wide locking of pull requests to get stuck. The threads block until the cluster locks are acquired.

      These threads are participating in transactions that have DB connections open. This can lead to nodes' DB connection pools getting exhausted.

      Sample log excerpts showing DB pool exhaustion and the stack for threads holding DB connections:

      2025-01-28 22:18:56,101 WARN  [ssh-auth]  c.a.s.i.jdbc.ConnectionTracker The database pool is exhausted. Stacks for the threads holding connections follow:
      2025-01-28 22:18:56,104 WARN  [ssh-auth]  c.a.s.i.jdbc.ConnectionTracker Stack trace for http-nio-127.0.0.1-7991-exec-50
      com.atlassian.stash.internal.util.StackException: Stack trace for http-nio-127.0.0.1-7991-exec-50
      	at java.base@17.0.12/jdk.internal.misc.Unsafe.park(Native Method)
      	at java.base@17.0.12/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341)
      	at com.hazelcast.spi.impl.AbstractInvocationFuture.manageParking(AbstractInvocationFuture.java:693)
      	at com.hazelcast.spi.impl.AbstractInvocationFuture.joinInternal(AbstractInvocationFuture.java:582)
      	at com.hazelcast.internal.locksupport.LockProxySupport.lock(LockProxySupport.java:67)
      	at com.hazelcast.internal.locksupport.LockProxySupport.lock(LockProxySupport.java:59)
      	at com.hazelcast.map.impl.proxy.MapProxyImpl.lock(MapProxyImpl.java:320)
      	at com.atlassian.beehive.hazelcast5.HazelcastMappedLock.lock(HazelcastMappedLock.java:39)
      	at com.atlassian.beehive.hazelcast5.HazelcastClusterLock.lock(HazelcastClusterLock.java:38)
      	at com.atlassian.stash.internal.concurrent.DefaultMonitoredClusterLockService$MonitoredClusterLock.lambda$lock$0(DefaultMonitoredClusterLockService.java:96)
      	at com.atlassian.stash.internal.concurrent.DefaultMonitoredClusterLockService$MonitoredClusterLock$$Lambda$2972/0x00007fb99567bd90.perform(Unknown Source)
      	at com.atlassian.stash.internal.concurrent.DefaultMonitoredClusterLockService$MonitoredClusterLock.trackWaiting(DefaultMonitoredClusterLockService.java:158)
      	at com.atlassian.stash.internal.concurrent.DefaultMonitoredClusterLockService$MonitoredClusterLock.lock(DefaultMonitoredClusterLockService.java:95)
      	at com.atlassian.stash.internal.concurrent.ClusterMappedLock.withLock(ClusterMappedLock.java:47)
      	at com.atlassian.stash.internal.concurrent.ClusterPullRequestLock.withLock(ClusterPullRequestLock.java:25)
      	at com.atlassian.stash.internal.concurrent.ClusterPullRequestLock.withLock(ClusterPullRequestLock.java:30)
      	at com.atlassian.stash.internal.pull.HazelcastPullRequestStateGuard.processIfUpdated(HazelcastPullRequestStateGuard.java:40)
      	at com.atlassian.stash.internal.pull.comment.DelegatingCommentUpdateProcessor.maybeProcess(DelegatingCommentUpdateProcessor.java:23)
      	at jdk.internal.reflect.GeneratedMethodAccessor1111.invoke(Unknown Source)
      <trimmed>
      

      Feature request

      Add improvements/handling to methods related to locking pull requests so that they do not get blocked and lead to DB pool exhaustion (e.g. implement timeouts)

              Unassigned Unassigned
              jmariano@atlassian.com JP Mariano
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: