Uploaded image for project: 'Bitbucket Data Center'
  1. Bitbucket Data Center
  2. BSERV-13229

Performance degradation in permissions cache

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Duplicate
    • Icon: Medium Medium
    • None
    • 7.17.6
    • Database - Permissions

      Issue Summary

      Post bug fix for BSERV-13141, there is an intermittent performance delay in loading the dashboard (or any page that needs a permissions check). This is caused by the first permissions lookup taking longer than expected.

      In the example issue triggering this bug, the issue was noticed loading the dashboard by an admin with a large group and permission set.

      13776.5	GET /dashboard HTTP/1.1	username
      12363.9	GET /dashboard HTTP/1.1	username 
      

      Profiler logs show that the time spent was finding groups for the user (EffectivePermissionDao.findByGroup):

      2022-03-29 20:53:59,918 | http-nio-7101-exec-95 | *IYTOKJx1253x9793273x2 | username | bqhlaz
      [12363.9ms] - "GET /dashboard HTTP/1.1"
       [1.3ms] - ApplicationUser com.atlassian.bitbucket.user.UserService.getUserById(int)
       [12193.7ms] - Page com.atlassian.bitbucket.user.RecentlyAccessedRepositoriesService.findByCurrentUser(Permission,PageRequest)
        [12193.7ms] - Page com.atlassian.stash.internal.user.RepositoryAccessDao.findRecentRepositories(int,PageRequest,Predicate)
         [2.1ms] - Iterable com.atlassian.stash.internal.user.InternalUserService.getGroupsByUser(String)
         [11604.5ms] [count: 65, avg: 178.5ms] - Page com.atlassian.stash.internal.user.EffectivePermissionDao.findByGroup(String,PageRequest)
         [1.7ms] - Iterable com.atlassian.stash.internal.user.InternalUserService.getGroupsByUser(String)
       [8.4ms] - Page com.atlassian.bitbucket.dashboard.DashboardService.suggestPullRequests(SuggestPullRequestsRequest)
      
      

      The previous two bugs inline with this ticket are BSERV-13056 and BSERV-13141.

      Steps to Reproduce

      1. Create 55k repositories.
      2. Assign group with admin privilege to each repository.

      Expected Results

      A runtime of a few hundred milliseconds or, at most, a second, resulting in a fast loading UI:

      2022-02-09 15:59:37,289 | http-nio-7990-exec-5 | @CCMM57x959x54x0 | admin | 1kq6332
      [1395.4ms] - "GET /dashboard HTTP/1.1"
       [1079.4ms] - Page com.atlassian.bitbucket.user.RecentlyAccessedRepositoriesService.findByCurrentUser(Permission,PageRequest)
        [1079.4ms] - Page com.atlassian.stash.internal.user.RepositoryAccessDao.findRecentRepositories(int,PageRequest,Predicate)
         [1.8ms] - Iterable com.atlassian.stash.internal.user.InternalUserService.getGroupsByUser(String)
         [911.6ms] [count: 31, avg: 29.4ms] - Page com.atlassian.stash.internal.user.EffectivePermissionDao.findByGroup(String,PageRequest)
       [24.6ms] - Page com.atlassian.bitbucket.dashboard.DashboardService.suggestPullRequests(SuggestPullRequestsRequest)
        [24.6ms] - void com.atlassian.stash.internal.repository.RepositoryActivityDao.findRecentPushesForUser(int,Date,SatiableConsumer)
      

      Actual Results

      A runtime in the tens of thousands of milliseconds, resulting in a longer wait for application response.

      2022-03-29 21:11:45,716 | http-nio-7101-exec-106 | *IYTOKJx1271x9855419x0 | username | bqhlaz
      [13776.5ms] - "GET /dashboard HTTP/1.1"
       [1.5ms] - ApplicationUser com.atlassian.bitbucket.user.UserService.getUserById(int)
       [13629.2ms] - Page com.atlassian.bitbucket.user.RecentlyAccessedRepositoriesService.findByCurrentUser(Permission,PageRequest)
        [13629.2ms] - Page com.atlassian.stash.internal.user.RepositoryAccessDao.findRecentRepositories(int,PageRequest,Predicate)
         [1.2ms] - Iterable com.atlassian.stash.internal.user.InternalUserService.getGroupsByUser(String)
         [13044.8ms] [count: 65, avg: 200.7ms] - Page com.atlassian.stash.internal.user.EffectivePermissionDao.findByGroup(String,PageRequest)
       [8.1ms] - Page com.atlassian.bitbucket.dashboard.DashboardService.suggestPullRequests(SuggestPullRequestsRequest)
      
      

      After initial delay, subsequent requests are fast.

      Workaround

      • The issue can be worked around by restricting the number of repositories a group or user has explicit permissions to. One of the ways to do this is to roll up a group's permissions from the repository to the project level.

              Unassigned Unassigned
              mwilliams@atlassian.com Mark A.
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: