Creating a pull request with nested groups enabled takes a very long time

It takes a long time for Bitbucket Server to resolve reviewers for a Pull Request when said reviewers are at the bottom of a structure of nested groups.

Environment

User directory with lots of groups and users that are at the bottom of nested structure of those groups.

Steps to Reproduce

1. Have a complicated structure of user directory with multiple levels of nested groups
2. Add a group with "read" permissions to a project
3. Create a pull request with couple users from that group as reviewers

Expected Results

Pull request is created immediately

Actual Results

Pull request takes up to 10 minutes to create, depending on number of groups

Notes

Whenever Bitbucket Server resolves reviewers, it validates those user permissions.
Users must:

• have permission to user Bitbucket Server (be a licensed user)
• have permission to access the repository pull request is being created in

In order to do that, because of the nested groups structure Bitbucket Server produces a large number of calls to HibernateMembershipDao#search method that fetches users, groups and membership information from the database. When the amount of results this method fetches from the database is large enough, it pollutes Hibernate context to the point when auto flush takes a lot of time. In order to avoid that, Bitbucket Server configured to run this method with transaction that is in "read-only" mode so Hibernate can skip context flushing as there are no changes being made to those entities. However, during PR creation, the method runs in context of outer transaction, which is opened by DefaultPullRequestService#create method and this transaction is "read-write" so this mitigation doesn't work.

Workaround

Max out users, groups and permissions caches as well as page size for permissions by adding the following into bibucket.properties file:

cache.com.atlassian.crowd.model.user.InternalUser.max=30000
permissions.cache.users.max=30000
cache.com.atlassian.crowd.model.user.InternalUser.attributes.max=30000
cache.com.atlassian.crowd.model.user.InternalUser.credentialRecords.max=30000
cache.com.atlassian.crowd.model.user.InternalUserAttribute.max=30000
cache.com.atlassian.crowd.model.group.InternalGroup.max=10000
permissions.cache.groups.max=30000
cache.query.groupMemberships.max=30000
cache.query.groupMemberships.ttl=2400
cache.query.permissions.max=30000
page.max.granted.permissions=20000

The numbers must be adjusted accordingly with respect to amount of users and groups in the system.

The workaround seems to bring the time down to 1 minute.