Adaptive throttling of SCM hosting operations

SCM hosting operations (e.g., git pull, etc.) have been throttled since the first release of Bitbucket Server. Git processes can be memory and CPU hungry, and if too many are run at once it can degrade performance of the instance for all other (non hosting) requests. So in Bitbucket Server releases up to and including 4.10, throttling was achieved by allocating a fixed number of tickets, and each hosting operation would have to acquire a ticket before it could proceed. Hosting operations finding no tickets available had to queue until one was available and would time out if it queued for too long. The number of available tickets was configured with the throttle.resource.scm-hosting property.

The default value of throttle.resource.scm-hosting was 1.5 times the number of CPU cores available in the machine. This was a reasonable default for many instances, but for some instances with ample hardware resources and a hosting load consisting of many small, light Git operations, it can lead to under-utilization of the server hardware, and users might be waiting longer for their clones than they'd like or need to.

4.11 introduces new throttling approach for SCM hosting operations that adapts to the stress the machine is under. It has the following characteristics:

• It allows a variable range of tickets values depending on how close current CPU usage is to a target CPU usage (defaults to 75% CPU usage). This can be changed.
• The default min and max of the ticket range are [1 ticket per CPU core, 12 tickets per CPU core]. These defaults can be changed.
• Every 5 seconds it resamples the CPU usage and recalculates how many tickets within the range can be supported.
• CPU readings are smoothed so as to not respond too suddenly to CPU spikes and overshoot/undershoot the optimum number of tickets..
• Examines the total physical memory on the machine and determines a maximum ticket number that the machine can safely support given an estimate of how much memory a hosting operation consumes, how much memory the Bitbucket Server service needs and how much Elasticsearch needs (if Elasticsearch is running locally).
• Adaptive throttling is enabled by default but may revert to the pre-4.11 throttling strategy as described below.
• The property throttle.resource.scm-hosting is deprecated and is replaced with a number of other properties to configure adaptive throttling and the original fixed throttling 

Fixed throttling is still supported. It will be activated for SCM hosting if:

• A fixed number of tickets e.g. throttle.resource.scm-hosting=25 was explicitly configured in bitbucket.properties.
• The fixed strategy is configured explicitly e.g. throttle.resource.scm-hosting.strategy=fixed and throttle.resource.scm-hosting.fixed.limit=25.
• The adaptive throttling configuration is invalid in some way.
• The total memory of the machine is so limited that the entire tickets range is unsafe.

Properties used by adaptive throttling and their defaults (see the Resource throttling section in Bitbucket Server config properties for a longer description of each):

# Controls how long threads will wait for SCM hosting operations to complete when the system is already
# running the maximum number of SCM hosting operations.
#
# This value is in **seconds**.
throttle.resource.scm-hosting.timeout=300
 
# Specifies the strategy for throttling SCM hosting operations. Possible values are 'adaptive' and 'fixed'.
throttle.resource.scm-hosting.strategy=adaptive
 
# When the adaptive strategy is enabled for throttling SCM hosting operations, this sets the lower limit on
# the number of SCM hosting operations, meaning pushes and pulls over HTTP or SSH, which may be running concurrently.
throttle.resource.scm-hosting.adaptive.limit.min=1*${scaling.concurrency}
 
# When the adaptive strategy is enabled for throttling SCM hosting operations, this sets the upper limit on
# the number of SCM hosting operations, meaning pushes and pulls over HTTP or SSH, which may be running concurrently.
# This is intended primarily to prevent pulls, which can be very memory-intensive, from exhausting a server's resources.
throttle.resource.scm-hosting.adaptive.limit.max=12*${scaling.concurrency}
 
# When the adaptive strategy is enabled for throttling SCM hosting operations, this sets the target CPU utilisation for
# the machine (across all processors) which the system takes into consideration when calculating the current throttling
# limit.
# This must be a value between 0.0 and 1.0 and is a percentage of the total available CPU power across all cores
throttle.resource.scm-hosting.adaptive.cpu.target=0.75
 
# When the fixed strategy is enabled for throttling SCM hosting operations, this limits the number of SCM
# hosting operations, meaning pushes and pulls over HTTP or SSH, which may be running concurrently. This is intended
# primarily to prevent pulls, which can be very memory-intensive, from exhausting a server's resources. There is limited
# support for mathematical expressions; +,-,*,/ and () are supported.
throttle.resource.scm-hosting.fixed.limit=1.5*${scaling.concurrency}