Issue Summary

When several pull requests are queued in a merge queue, Bitbucket builds a temporary merge queue branch for each one. Each of those branches is built on top of the queue branch of the pull request ahead of it, so that every pull request is validated against the state the target branch will be in once the pull requests ahead of it have merged.

A consequence of this is that when the pull request at the front of the queue merges, the target branch moves to exactly the commit that the next pull request's queue branch was already built on top of. The remaining queue branches are therefore still valid, and are expected to be reused unchanged.

Before the merge queue reuses a queue branch it checks whether the pull request's target branch has moved since that branch was built. That check uses the pull request's own toHash, which is only updated when the pull request is rescoped. Rescoping happens asynchronously after the merge completes, and the merge queue can re-evaluate the remaining pull requests before the rescope has been applied. When that happens the merge queue reads a toHash that still refers to the pre-merge target commit, concludes that the target branch has moved on, and rebuilds the queue branches, even though the resulting branches contain the same changes as the ones they replace.

Each rebuild publishes another pr:merge_queue:branch_prepared webhook with a new commit hash. Integrations that use these webhooks to drive builds will see what looks like a new revision to build, and integrations that cancel a running build when a new revision arrives will abort builds that were about to pass.

The merges themselves are not affected. Pull requests still merge in the correct order and the merged result is correct. The impact is unnecessary Git work on the server, unnecessary webhook traffic, and wasted CI capacity.

The merge queue re-evaluates a queue in response to a build status being posted against one of its commits. Integrations that post build statuses frequently therefore re-evaluate the queue frequently, and each re-evaluation that lands before the rescope has been applied will rebuild the remaining queue branches. Because each rebuild produces a new commit for the integration to build and report on, this can become self sustaining for as long as the queue is busy.

Because it depends on whether the rescope is applied before or after the merge queue re-evaluates the queue, the behaviour is intermittent. It is more likely on instances where rescoping takes longer to complete, for example:

• a large number of open pull requests targeting the same branch, since all of them are rescoped when that branch moves
• repositories large enough that the Git operations involved in rescoping take a noticeable amount of time
• a busy instance, or one where Git operations are already slow

Steps to Reproduce

1. Use a Bitbucket Data Center instance with a merge queue enabled on a branch, for example the default branch.
2. Create a webhook subscribed to the merge queue branch prepared and merge queue branch deleted events, and record every event it receives.
3. Enable required builds with a merge queue execution type for the target branch.
4. Create three pull requests targeting that branch and add all three to the merge queue, so that they are queued in order.
5. Allow the first pull request in the queue to pass its checks and merge.
6. Immediately after the merge completes, post a build status against the head commit of the second pull request's merge queue branch, using POST /rest/api/1.0/projects/{projectKey}/repos/{repositorySlug}/commits/{commitId}/builds. This forces the merge queue to re-evaluate the queue while the rescope may still be in progress, which is what a continuous integration system does automatically.
7. Compare the merge queue branch commit hashes reported for the second and third pull requests before and after the first pull request merged.
8. Confirm that no commits were pushed to the source branches of the second and third pull requests during this time, so that any rebuild of their queue branches cannot be attributed to a change in the pull request itself.

Note that this depends on timing, so it will not occur on every attempt. It is most readily observed on an instance that already has a large number of open pull requests targeting the same branch, which increases the time taken for rescoping to complete.

Expected Results

After the first pull request merges:

• The second and third pull requests keep their existing merge queue branches, because those branches were already built on top of the commit the target branch has just moved to.
• No further pr:merge_queue:branch_prepared webhook is published for the second or third pull request.
• Builds already running against those queue branches remain valid and are allowed to finish.
• A queue branch is only rebuilt when something has genuinely changed, such as a new commit pushed to the source branch, or the pull request ahead of it in the queue being removed or rebuilt.

Actual Results

After the first pull request merges:

• The merge queue branches for the second and third pull requests are rebuilt, and are reported with new commit hashes even though they contain the same changes.
• An additional pr:merge_queue:branch_prepared webhook is published for each of them.
• The rebuilds happen without any corresponding change to the pull requests themselves. No commits were pushed to their source branches, and their diffs are unchanged.
• Integrations that treat a new commit hash as a new revision start a new build, and integrations that cancel the previous build when a new revision arrives abort builds that were already close to passing.
• The rebuild can cascade, so rebuilding one queue branch causes the queue branch behind it to be rebuilt as well.

No error is reported for the unnecessary rebuild itself. Enabling debug logging for the merge queue will show the queue being processed and pull requests being merged, but it records nothing about a queue branch being rebuilt, so the rebuilds cannot be confirmed from the Bitbucket log files. The pull requests merge successfully and in the correct order, so the only symptoms available are the extra webhooks, the changed commit hashes, and the resulting CI churn.

Errors relating to rescoping may be present in the log files, for example Git command timeouts while rescoping. Those are not specific to merge queues and are not caused by this behaviour, so they should not be read as evidence of it. They do however make it more likely, because anything that delays rescoping widens the window in which a re-evaluation can read the pre-merge value.

Workaround

Currently there is no known workaround for this behavior. A workaround will be added here when available.