Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-13229

Performance degradation in permissions cache

    XMLWordPrintable

Details

    • Bug
    • Resolution: Duplicate
    • Medium
    • None
    • 7.17.6
    • Database - Permissions

    Description

      Issue Summary

      Post bug fix for BSERV-13141, there is an intermittent performance delay in loading the dashboard (or any page that needs a permissions check). This is caused by the first permissions lookup taking longer than expected.

      In the example issue triggering this bug, the issue was noticed loading the dashboard by an admin with a large group and permission set. 

      13776.5	GET /dashboard HTTP/1.1	username
12363.9	GET /dashboard HTTP/1.1	username

      Profiler logs show that the time spent was finding groups for the user (EffectivePermissionDao.findByGroup):

      2022-03-29 20:53:59,918 | http-nio-7101-exec-95 | *IYTOKJx1253x9793273x2 | username | bqhlaz
[12363.9ms] - "GET /dashboard HTTP/1.1"
 [1.3ms] - ApplicationUser com.atlassian.bitbucket.user.UserService.getUserById(int)
 [12193.7ms] - Page com.atlassian.bitbucket.user.RecentlyAccessedRepositoriesService.findByCurrentUser(Permission,PageRequest)
  [12193.7ms] - Page com.atlassian.stash.internal.user.RepositoryAccessDao.findRecentRepositories(int,PageRequest,Predicate)
   [2.1ms] - Iterable com.atlassian.stash.internal.user.InternalUserService.getGroupsByUser(String)
   [11604.5ms] [count: 65, avg: 178.5ms] - Page com.atlassian.stash.internal.user.EffectivePermissionDao.findByGroup(String,PageRequest)
   [1.7ms] - Iterable com.atlassian.stash.internal.user.InternalUserService.getGroupsByUser(String)
 [8.4ms] - Page com.atlassian.bitbucket.dashboard.DashboardService.suggestPullRequests(SuggestPullRequestsRequest)

      The previous two bugs inline with this ticket are BSERV-13056 and BSERV-13141.

      Steps to Reproduce

      1. Create 55k repositories.
      2. Assign group with admin privilege to each repository.

      Expected Results

      A runtime of a few hundred milliseconds or, at most, a second, resulting in a fast loading UI:

      2022-02-09 15:59:37,289 | http-nio-7990-exec-5 | @CCMM57x959x54x0 | admin | 1kq6332
[1395.4ms] - "GET /dashboard HTTP/1.1"
 [1079.4ms] - Page com.atlassian.bitbucket.user.RecentlyAccessedRepositoriesService.findByCurrentUser(Permission,PageRequest)
  [1079.4ms] - Page com.atlassian.stash.internal.user.RepositoryAccessDao.findRecentRepositories(int,PageRequest,Predicate)
   [1.8ms] - Iterable com.atlassian.stash.internal.user.InternalUserService.getGroupsByUser(String)
   [911.6ms] [count: 31, avg: 29.4ms] - Page com.atlassian.stash.internal.user.EffectivePermissionDao.findByGroup(String,PageRequest)
 [24.6ms] - Page com.atlassian.bitbucket.dashboard.DashboardService.suggestPullRequests(SuggestPullRequestsRequest)
  [24.6ms] - void com.atlassian.stash.internal.repository.RepositoryActivityDao.findRecentPushesForUser(int,Date,SatiableConsumer)

      Actual Results

      A runtime in the tens of thousands of milliseconds, resulting in a longer wait for application response.

      2022-03-29 21:11:45,716 | http-nio-7101-exec-106 | *IYTOKJx1271x9855419x0 | username | bqhlaz
[13776.5ms] - "GET /dashboard HTTP/1.1"
 [1.5ms] - ApplicationUser com.atlassian.bitbucket.user.UserService.getUserById(int)
 [13629.2ms] - Page com.atlassian.bitbucket.user.RecentlyAccessedRepositoriesService.findByCurrentUser(Permission,PageRequest)
  [13629.2ms] - Page com.atlassian.stash.internal.user.RepositoryAccessDao.findRecentRepositories(int,PageRequest,Predicate)
   [1.2ms] - Iterable com.atlassian.stash.internal.user.InternalUserService.getGroupsByUser(String)
   [13044.8ms] [count: 65, avg: 200.7ms] - Page com.atlassian.stash.internal.user.EffectivePermissionDao.findByGroup(String,PageRequest)
 [8.1ms] - Page com.atlassian.bitbucket.dashboard.DashboardService.suggestPullRequests(SuggestPullRequestsRequest)

      After initial delay, subsequent requests are fast.

      Workaround

      • The issue can be worked around by restricting the number of repositories a group or user has explicit permissions to. One of the ways to do this is to roll up a group's permissions from the repository to the project level.

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. BSERV-13582 Poor permissions cache performance leads to slow authentication/authorization

          • High - High priority issues
          • Closed
          is cloned from

          Bug - A problem which impairs or prevents the functions of the product. BSERV-13141 Performance degradation in permissions cache

          • Low - Low priority issues
          • Closed
          is related to

          Bug - A problem which impairs or prevents the functions of the product. BSERV-13056 Performance regression of methods that compute user permissions

          • Highest - Our top priority issues
          • Closed
          relates to

          Bug - A problem which impairs or prevents the functions of the product. BSERV-13261 Git throws an intermittent 500 CacheException error when cloning

          • Low - Low priority issues
          • Closed
          causes

          PS-90684 Loading...

          mentioned in

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Page Loading...

          Activity

            People

              Unassigned Unassigned
              mwilliams@atlassian.com Mark A.
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: