-
Bug
-
Resolution: Fixed
-
High
-
6.0.0, (60)
6.1.0, 6.0.1, 6.2.0, 6.0.2, 6.1.1, 6.0.3, 6.1.2, 6.0.4, 6.1.3, 6.3.0, 6.0.5, 6.1.4, 6.2.1, 6.0.6, 6.1.5, 6.2.2, 6.3.1, 6.4.0, 6.2.3, 6.3.2, 6.5.0, 6.0.7, 6.1.6, 6.2.4, 6.3.3, 6.4.1, 6.6.0, 6.3.4, 6.4.2, 6.5.1, 6.1.7, 6.0.8, 6.0.9, 6.2.5, 6.5.2, 6.0.10, 6.1.8, 6.2.6, 6.3.5, 6.4.3, 6.7.0, 6.6.1, 6.6.2, 6.7.1, 6.7.2, 6.8.0, 6.8.1, 6.9.0, 6.10.0, 6.0.11, 6.1.9, 6.2.7, 6.3.6, 6.4.4, 6.5.3, 6.6.3, 6.7.3, 6.8.2, 6.9.1, 6.10.1
-
9
-
Severity 2 - Major
-
3
-
Issue Summary
If users, groups and membership caches are cold or have insufficient size (a required size for membership cache can be well into 500 000 entries mark for decent directory sizes), it is virtually impossible to add a branch permission as UI times out while the application checks all the found users for required permissions.
Steps to Reproduce
- Star a LDAP server
- Import the users and groups from that dataset: tree.ldif.zip
- Synchronise Bitbucket with that LDAP server.
- Create a project
- Create a repository
- Add write permissions for groups: group.2132849, group.2137710, group.2138629, group.8126700
- Go to "Branch permission"
- Click "Add permission"
- Start typing (manually typing, no copy-pasting): cn 203 (full username - cn 2034568)
Expected Results
User presented with a list of user which names start with "cn 203"
Actual Results
Timeout after one minute.
Workaround
For provided dataset bumping permission cache, group permission cache, user cache, group cache is inefficient if nested groups are enabled. Disabling nested groups makes permissions caches relatively effective.
For nested groups case, for provided dataset, cache.query.groupMemberships.max had to be set into value well above 100 000 (I had it at 500 000).
The root cause of the issue is that it takes a query per-user-per-permission check to get the groups user belongs to (even more if nested groups are enabled).
See also jProfiler snapshot: another-users-ang-groups-failure.jps
- is related to
-
BSERV-11699 Only cache permissions for groups that actually have permissions granted
- Closed
- mentioned in
-
Page Failed to load
-
Page Failed to load
-
Page Failed to load
-
Page Failed to load
-
Page Loading...
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
| Form Name | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
Since this issue was raised we've made some improvements to the performance of permissions calculations. The below shows the request latency for various versions of Bitbucket, with 7.6 used as a baseline (it is our oldest currently supported LTS). Some changes were made in 7.14 thru 7.16, with a larger set of improvements being made in 8.7 and 8.8.
All requests, as of 8.8, are coming in under the 30 second timeout such that the UI no longer reports the timeout/error.
The numbers show request latency for user search for pattern "cn 203" using the dataset mentioned in the description. For all tests relevant caches have been tuned via the bitbucket.properties file, specifically tuning for 50,000 users and 20,000 groups. The dataset contains 72,458 users and 17,018 groups.
The "max" time is the latency for a cold cache; the test was run after restarting the Bitbucket instance to clear caches. The subsequent requests are for warm caches. The test also has some concurrency, with 10 request threads being used to simulate some load.
It is worth pointing out, in Bitbucket 8.8 we are making setting of these caches easier with two new public/documented properties added:
Advice for tuning these will be found in the "Scaling Bitbucket Server" document for versions 8.8 onwards. It is recommended these be tuned for any instance with more than 4000 users.