-
Suggestion
-
Resolution: Unresolved
-
None
-
None
-
6
-
Context
It is possible that clustering issues (e.g. split-brain scenario) may lead to threads calling methods on cluster-wide locking of pull requests to get stuck. The threads block until the cluster locks are acquired.
These threads are participating in transactions that have DB connections open. This can lead to nodes' DB connection pools getting exhausted.
Sample log excerpts showing DB pool exhaustion and the stack for threads holding DB connections:
2025-01-28 22:18:56,101 WARN [ssh-auth] c.a.s.i.jdbc.ConnectionTracker The database pool is exhausted. Stacks for the threads holding connections follow: 2025-01-28 22:18:56,104 WARN [ssh-auth] c.a.s.i.jdbc.ConnectionTracker Stack trace for http-nio-127.0.0.1-7991-exec-50 com.atlassian.stash.internal.util.StackException: Stack trace for http-nio-127.0.0.1-7991-exec-50 at java.base@17.0.12/jdk.internal.misc.Unsafe.park(Native Method) at java.base@17.0.12/java.util.concurrent.locks.LockSupport.park(LockSupport.java:341) at com.hazelcast.spi.impl.AbstractInvocationFuture.manageParking(AbstractInvocationFuture.java:693) at com.hazelcast.spi.impl.AbstractInvocationFuture.joinInternal(AbstractInvocationFuture.java:582) at com.hazelcast.internal.locksupport.LockProxySupport.lock(LockProxySupport.java:67) at com.hazelcast.internal.locksupport.LockProxySupport.lock(LockProxySupport.java:59) at com.hazelcast.map.impl.proxy.MapProxyImpl.lock(MapProxyImpl.java:320) at com.atlassian.beehive.hazelcast5.HazelcastMappedLock.lock(HazelcastMappedLock.java:39) at com.atlassian.beehive.hazelcast5.HazelcastClusterLock.lock(HazelcastClusterLock.java:38) at com.atlassian.stash.internal.concurrent.DefaultMonitoredClusterLockService$MonitoredClusterLock.lambda$lock$0(DefaultMonitoredClusterLockService.java:96) at com.atlassian.stash.internal.concurrent.DefaultMonitoredClusterLockService$MonitoredClusterLock$$Lambda$2972/0x00007fb99567bd90.perform(Unknown Source) at com.atlassian.stash.internal.concurrent.DefaultMonitoredClusterLockService$MonitoredClusterLock.trackWaiting(DefaultMonitoredClusterLockService.java:158) at com.atlassian.stash.internal.concurrent.DefaultMonitoredClusterLockService$MonitoredClusterLock.lock(DefaultMonitoredClusterLockService.java:95) at com.atlassian.stash.internal.concurrent.ClusterMappedLock.withLock(ClusterMappedLock.java:47) at com.atlassian.stash.internal.concurrent.ClusterPullRequestLock.withLock(ClusterPullRequestLock.java:25) at com.atlassian.stash.internal.concurrent.ClusterPullRequestLock.withLock(ClusterPullRequestLock.java:30) at com.atlassian.stash.internal.pull.HazelcastPullRequestStateGuard.processIfUpdated(HazelcastPullRequestStateGuard.java:40) at com.atlassian.stash.internal.pull.comment.DelegatingCommentUpdateProcessor.maybeProcess(DelegatingCommentUpdateProcessor.java:23) at jdk.internal.reflect.GeneratedMethodAccessor1111.invoke(Unknown Source) <trimmed>
Feature request
Add improvements/handling to methods related to locking pull requests so that they do not get blocked and lead to DB pool exhaustion (e.g. implement timeouts)
- relates to
-
ACE-7121 Loading...