Mirror stops syncing all repositories while some other large repository is being synced

Issue Summary

When a large repository is being synced during initial sync (i.e. when a repository is created on primary) on a mirror, changes in other repositories stop syncing during that timeframe. Once the large repository finishes syncing, the other repositories resume syncing. This can also happen when an incremental sync process of a repository takes very long.

This is reproducible on Data Center: yes

Steps to Reproduce

1. Setup Bitbucket primary instance and a single mirror
2. Create a project with 1 or more repositories.
3. Add this project to the mirrored list and wait for all the repositories to be synced.
4. Push some changes to one of the mirrored repositories says Repo-1 and they should be synced successfully on the mirror.
5. On primary instance, import some medium size repository e.g. https://github.com/facebook/react.git in the mirrored project and wait for the import process to finish.
6. As soon as the repository import finishes, keep pushing some new changes in the Repo-1 for more than 2 minutes. These changes should be pushed while the imported repository is still being synced on the mirror.
7. Run git ls-remote for Repo-1 against the mirror.

Expected Results

The newly pushed changes in Repo-1 should be present in the ls-remote output after a couple of seconds since they pushed.

Actual Results

The newly pushed changes in Repo-1 are not present in the ls-remote output until the newly imported repository finishes syncing on the mirror.

Workaround

Set plugin.mirroring.lock.harvest.interval=86400 which is equivalent to 1 day. This will reduce the probability of the issue from every minute to once a day.