Issue Summary

Bitbucket's dashboard has a feature where when a user pushes a branch a suggestion will be displayed offering to create a pull request from the recently pushed branch. The code behind this will look at all recent pushed, by that user, and offer up to three branches as suggestions.

In the case where the user pushes a new existing repository which happens to have a large number of branches this creates a lot of potential candidates for suggestions. Bitbucket has an internal limit such that even if you were to push thousands of refs (branches or tags) it will only evaluate the last 200 as potential candidates.

There are some inefficiencies around how this is done however, and it can lead to slow responses from the /rest/ui/latest/dashboard/pull-request-suggestions REST API.

The problem described here happens when a call to RefService.resolveRef() becomes slow. Take this example from the profiling logs:

2024-08-21 04:05:44,529 | http-nio-7990-exec-15 | @FHH7MTx245x2532x0 | admin | x05yvp
[1185.2ms] - "GET /rest/ui/latest/dashboard/pull-request-suggestions HTTP/1.1"
 [1180.5ms] - Page com.atlassian.bitbucket.dashboard.DashboardService.suggestPullRequests(SuggestPullRequestsRequest)
  [1180.5ms] - void com.atlassian.stash.internal.repository.RepositoryActivityDao.findRecentPushesForUser(int,Date,SatiableConsumer)
   [3.8ms] - Branch com.atlassian.bitbucket.repository.RefService.getDefaultBranch(Repository)
    [3.7ms] - Branch com.atlassian.stash.internal.scm.git.mesh.RpcRefClient.resolveDefaultBranch(Repository,Builder)
   [890.0ms] [count: 200, avg: 4.5ms] - Ref com.atlassian.bitbucket.repository.RefService.resolveRef(ResolveRefRequest)
    [887.0ms] [count: 200, avg: 4.4ms] - Map com.atlassian.stash.internal.scm.git.mesh.RpcRefClient.resolveRefs(Repository,Builder)

Specifically this row:

[890.0ms] [count: 200, avg: 4.5ms] - Ref com.atlassian.bitbucket.repository.RefService.resolveRef(ResolveRefRequest)

That says Bitbucket evaluated 200 candidate pushed refs for possible suggestion, and on average resolveRef() took 4.5ms to complete. So the total time to respond was about 1 second.

However it is possible for resolveRef to be slower. If for example it took 500ms to respond then the entire operation would take 200 * 500ms = 100 seconds to complete.

This issue does not deal with the cause of resolveRef() being slow; here we only discuss how the Dashboard PR suggestions can be impacted by such a problem, and what possible improvements could be made. Typical causes of resolveRefs() being being slow include:

• Large volume of newly pushed refs that are still loose (i.e. haven't been packed yet)
• Slow filesystem that hosts repositories (e.g. slow NFS)

When new/updated refs are not packed, even on well performing NFS server `git for-each-ref` may take a second or more for 10,000 refs; scaling linearly as ref count increases.

Suggested improvements

Avoid resolveRefs() for tags
Currently SuggestionCandidateConsumer calls resolveRef() for tags, so if a user pushes a lot of tags Bitbucket has to work through them until it gets to a branch. It is possible to detect that a ref is a tag before calling resolveRef().

Add a timeout
A timeout should be added to the pull-request-suggestions REST endpoint. The result does not need to be deterministic, if the system can't build a list of suggestions in a reasonable amount of time (e.g. 10 seconds) it is reasonable to return an empty list.

Expected Results

The REST API /rest/ui/latest/dashboard/pull-request-suggestions responds quickly (e.g. a few seconds at most)

Actual Results

The REST API /rest/ui/latest/dashboard/pull-request-suggestions responds slowly.

Workaround

Currently there is no known workaround for this behavior. A workaround will be added here when available.

However the problem will resolve itself given time, at least until further pushes of large volumes of refs (branches/tags) occur. The pull request suggestions only look at pushes in the past 48 hours, so once a push falls beyond that horizon it will no longer be processed. Additionally, if the slow resolveRef() call is due to loose refs, when GC is automatically run refs will be packed, improving performance.