Bitbucket has a tuning property plugin.bitbucket-repository-ref-sync.threads that is described as follows:

# Controls the number of threads used for ref synchronisation. Higher values here will help synchronisation keep up
# with upstream updates, but may produce noticeable additional server load.
plugin.bitbucket-repository-ref-sync.threads=3

The value is not per node, but rather is per cluster. So for different cluster node counts, the per node number varies. The per node thread count is calculated by: ceil(value / node count). For example given the default value of 3:

I have the following three concerns about this behaviour:

1) It is not well documented
The "Configuration properties" documentation does not describe if this tunable is per cluster or per node. It is at least described in a support knowledge base article: https://support.atlassian.com/bitbucket-data-center/kb/forked-repository-does-not-get-synchronized-until-fork-syncing-is-turned-off-and-on-again/

2) It is inconsistent
This appears to be the only property for bucketed executor thread pool limits that has a thread pool sized per-cluster:

3) If provides poor scaling behaviour

The BucketedExecutor is a framework that enables background processing of tasks across the cluster. And is designed to allow more work to be done as the cluster grows in node count. Usually you'd expect linear scaling as node count grows incrementally, i.e. as node counts grow from 1 thru 5, total ref-sync threads grow 3, 6, 9, 12, 15. But instead it scales sub-linearly and even has a stepwise degradation when you go from 2 to 3 nodes, the sequence is 3, 4, 3, 4, 5

Note: This behaviour has been present in Bitbucket since Bitbucket/Stash 3.3.0.

Suggested fix

For existing versions of Bitbucket this behaviour should be documented in the "Configuration properties" documentation.

In a future minor or major version of Bitbucket we should consider moving this to a per-node number. I however do not suggest the change is backported to existing versions, this is potentially a large change in behaviour that is inappropriate for a bugfix release. It is something that should be called out in the release notes clearly. I say this because I see cases where customers have set this to (for example) 16. They have a 4 node cluster and want 4 threads per node, so set it to 16. Moving to a per node property means they go from 4 threads to 16 threads, potentially overloading their system, especially if the individual nodes are small.

Workaround

The plugin.bitbucket-repository-ref-sync.threads property can be set in bitbucket.properties to increase the ref-sync thread pool size. Because the value is divided across the cluster, it must be set to "desired_threads_per_node × number_of_nodes". Note that this requires a restart of all nodes to take effect, however can be done with a rolling (node by node) restart. A full cluster outage is not required.