Issue Summary

• Under concurrent git clone/fetch and push load, Bitbucket's HTTP port becomes permanently unresponsive requiring a proxy restart on the HTTP Port.
• The root cause is a deadlock in GrpcHostingClient where all Tomcat HTTP threads become permanently blocked in BidirectionalRemoteProcess.run():161 waiting on an unbounded LinkedBlockingQueue.take() with no timeout.
• Hook callbacks from Mesh cannot land on a free thread, causing the queue
  to never drain. The issue affects all tested versions (8.19.x, 9.4.x, 10.2.x) and is not
  mitigated by upgrading to JDK 21 or enabling virtual threads.

  Steps to Reproduce

1. Ensure Bitbucket is proxied via Apache httpd with ProxyPass to the HTTP port
2. Fire 300+ concurrent git clone/fetch operations against a repository
3. Simultaneously trigger push operations to the same repository (causing Mesh hook callbacks)
4. Observe the HTTP port. All operations will hang and the port becomes unresponsive

Expected Results

Bitbucket's HTTP port remains responsive under concurrent clone/fetch/push load.
Hook callbacks from Mesh are processed independently of the Tomcat HTTP thread pool,
preventing thread exhaustion.

Actual Results

All Tomcat HTTP threads become permanently blocked in BidirectionalRemoteProcess.run():161 waiting on LinkedBlockingQueue.take() for a Mesh hook callback that can never be delivered because no free threads remain to handle the inbound callback request. The HTTP port stops responding to all requests (git operations, REST API, UI) until Apache is restarted for the HTTP Port.

Confirmed across:

• Bitbucket 8.19.20 + JDK 17
• Bitbucket 9.4.21 + JDK 21
• Bitbucket 10.2.4 + JDK 21 + Tomcat 10.1.55 + virtual threads enabled

Thread dump shows all http-nio-exec threads stuck at:

"http-nio-7990-exec-1" #xx daemon prio=5 os_prio=0 tid=0x... nid=0x... waiting on condition
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for <0x...> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at com.atlassian.bitbucket.internal.mesh.hosting.AbstractBidirectionalHostingFragmentResponseObserver$BidirectionalRemoteProcess.run(AbstractBidirectionalHostingFragmentResponseObserver.java:161)
        at com.atlassian.bitbucket.internal.mesh.hosting.GrpcHostingClient.httpBackend(GrpcHostingClient.java:xxx)

Workaround

Configure Apache httpd to cap concurrent connections to Bitbucket at server.tomcat.threads.max
minus 5, reserving headroom for Mesh hook callbacks:

LoadModule proxy_module lib/httpd/modules/mod_proxy.so
LoadModule proxy_http_module lib/httpd/modules/mod_proxy_http.so
LoadModule proxy_balancer_module lib/httpd/modules/mod_proxy_balancer.so
LoadModule slotmem_shm_module lib/httpd/modules/mod_slotmem_shm.so
LoadModule lbmethod_byrequests_module lib/httpd/modules/mod_lbmethod_byrequests.so

<Proxy "balancer://bb">
    BalancerMember "http://localhost:7990" max=N
</Proxy>

ProxyPass "/bitbucket" "balancer://bb/bitbucket"
ProxyPassReverse "/bitbucket" "balancer://bb/bitbucket"

Formula: max = server.tomcat.threads.max - 5 (default server.tomcat.threads.max=20, so max=15)
This prevents all Tomcat threads from being consumed by git operations simultaneously,
ensuring hook callbacks from Mesh always have a free thread available.
Note: This workaround mitigates the symptom but does not fix the underlying root cause.