Type: Bug
Resolution: Fixed
Priority: High
Fix Version/s: 2.12.0
Affects Version/s: 1.0
Component/s: Enterprise
Labels:

Bug Fix Policy:
View Atlassian Server bug fix policy

getUsersWithPermission users UserService.findUsersByGroup to load the StashUsers for each group. It then calls getName() name on all of these users to get the usernames. Under the hood embedded crowd loads each user by name for the findUsersByGroup query if there are multiple directories and the user isn't in the first one. This can make getUsersWithPermission incredibly expensive (n+1 problem).

The LicenseService attempts to mitigate this by caching the licensed user count. However it maintains 5 separate caches requiring the licensed user count to be repopulated for each. The cache should be consolidated and repopulated in the background to minimise the impact on end users.

Workaround

The current workaround is to reorder the directories to have the largest directory first. When doing this you should be mindful of ~~STASH-3717~~ which can lead to users being locked out of Stash

was split into

CWD-3809 ApplicationServiceGeneric#searchNestedGroupRelationships performs n+1 queries in order to shadow users

Closed

BSERV-4396 Increase license service cache expiration

Closed

mentioned in: Page Failed to load; Page Failed to load; Page Failed to load; Page Loading...; Page Loading...

(2 mentioned in)

T. Alexander Lystad added a comment - 20/Feb/2014 7:54 AM

I did try the workaround mentioned in the description, without any significant impact

Thanks for the additional info.

T. Alexander Lystad added a comment - 20/Feb/2014 7:54 AM I did try the workaround mentioned in the description, without any significant impact Thanks for the additional info.

jhinch (Atlassian) added a comment - 20/Feb/2014 12:52 AM

Stash 2.11 isn't out yet (should be out next week). Unfortunately due to the extent of the change required to fix this bug we have delayed the fix until 2.12. Stash has approximately a 5 week release cycle. In terms of a workaround there is one currently documented in the description of this ticket. During the development of the fix I discovered that the cache expiration for the licensed user cache was very low (only 1 minute). I've created ~~STASH-4396~~ in order to track increasing that expiration. This fix can be backported to 2.11 (Most likely 2.11.1)

jhinch (Atlassian) added a comment - 20/Feb/2014 12:52 AM Stash 2.11 isn't out yet (should be out next week). Unfortunately due to the extent of the change required to fix this bug we have delayed the fix until 2.12. Stash has approximately a 5 week release cycle. In terms of a workaround there is one currently documented in the description of this ticket. During the development of the fix I discovered that the cache expiration for the licensed user cache was very low (only 1 minute). I've created STASH-4396 in order to track increasing that expiration. This fix can be backported to 2.11 (Most likely 2.11.1)

T. Alexander Lystad added a comment - 20/Feb/2014 12:16 AM - edited

Jason, when will you release 2.12?

Also, is there some workaround for this issue?

Currently I have to restart Stash multiple times a day.

T. Alexander Lystad added a comment - 20/Feb/2014 12:16 AM - edited Jason, when will you release 2.12? Also, is there some workaround for this issue? Currently I have to restart Stash multiple times a day.

jhinch (Atlassian) added a comment - 20/Feb/2014 12:05 AM

For 2.12 the underlying issue will be mitigated to a certain extent by:

Consolidating our caches
Only repopulating the licensed user cache in the background
Only ever having one thread repopulating the licensed user cache at any one time to minimise thrashing the hibernate L2 caches.

We will continue to monitor the issue and have raised ~~CWD-3809~~ to address the underlying performance issue.

jhinch (Atlassian) added a comment - 20/Feb/2014 12:05 AM For 2.12 the underlying issue will be mitigated to a certain extent by: Consolidating our caches Only repopulating the licensed user cache in the background Only ever having one thread repopulating the licensed user cache at any one time to minimise thrashing the hibernate L2 caches. We will continue to monitor the issue and have raised CWD-3809 to address the underlying performance issue.

T. Alexander Lystad added a comment - 24/Jan/2014 9:20 PM

Link: SSP-3296

T. Alexander Lystad added a comment - 24/Jan/2014 9:20 PM Link: SSP-3296

T. Alexander Lystad added a comment - 24/Jan/2014 9:19 PM

Ah, so I guess this is what is screwing us.

2014-01-24 11:11:51,203 | http-apr-7990-exec-22 | 671x952899x4 | TeamCityBuildServer | 1xrzail
[10156ms] - "GET /scm/VP/vismapayroll.git/info/refs HTTP/1.1"
[0ms] - com.atlassian.stash.internal.spring.security.StashAuthenticationFilter.requiresAuthentication
[109ms] - Authentication org.springframework.security.authentication.AuthenticationProvider.authenticate(Authentication)
[16ms] - attemptAuthentication - com.atlassian.stash.stash-auth-crowd-sso:crowdSsoAuthHandler
[0ms] - CaptchaTicket com.atlassian.stash.internal.user.CaptchaService.checkCaptcha(String,CaptchaResponse)
[0ms] - StashUser com.atlassian.stash.internal.user.CaptchaService.authenticateWithCaptcha(CaptchaTicket,UncheckedOperation)
[93ms] - attemptAuthentication - com.atlassian.stash.stash-authentication:crowdHttpAuthHandler
[93ms] - StashUser com.atlassian.stash.internal.user.CaptchaService.authenticateWithCaptcha(CaptchaTicket,UncheckedOperation)
[93ms] - StashUser com.atlassian.stash.user.UserService.authenticate(String,String)
[0ms] - InternalStashUser com.atlassian.stash.internal.user.StashUserDao.loadUser(User)
[0ms] - boolean com.atlassian.stash.internal.user.InternalPermissionService.hasGlobalPermission(StashUserAuthenticationToken,Permission)
[0ms] - Page com.atlassian.stash.user.UserService.findGroupsByUser(String,PageRequest)
[0ms] - Page com.atlassian.stash.internal.user.EffectivePermissionDao.findByGroup(Iterable,PageRequest)
[0ms] - Object com.atlassian.stash.user.SecurityService.doWithPermission(String,Permission,Operation)
[0ms] - Repository com.atlassian.stash.repository.RepositoryService.getBySlug(String,String)
[0ms] - InternalRepository com.atlassian.stash.internal.repository.RepositoryDao.getBySlug(String,String)
[0ms] - boolean com.atlassian.stash.user.PermissionService.isRepositoryAccessible(Repository)
[8641ms] - boolean com.atlassian.stash.scm.ScmRequestCheckService.checkActionAllowed(ScmRequest)
[8625ms] - KeyedMessage com.atlassian.stash.license.LicenseService.getStatus()
[8625ms] - int com.atlassian.stash.internal.user.InternalPermissionService.getCountOfUsersWithPermission(Permission)
[0ms] - Page com.atlassian.stash.internal.user.EffectivePermissionDao.findUsers(Permission,PageRequest,Predicate)
[0ms] - Page com.atlassian.stash.internal.user.EffectivePermissionDao.findGroups(Permission,PageRequest)
[0ms] - Page com.atlassian.stash.user.UserService.findUsersByGroup(String,PageRequest)
[0ms] - Set com.atlassian.stash.internal.user.StashUserDao.loadUsers(Iterable)
[8610ms] - Page com.atlassian.stash.user.UserService.findUsersByGroup(String,PageRequest)
[0ms] - Set com.atlassian.stash.internal.user.StashUserDao.loadUsers(Iterable)
[15ms] - Page com.atlassian.stash.user.UserService.findUsersByGroup(String,PageRequest)
[0ms] - Set com.atlassian.stash.internal.user.StashUserDao.loadUsers(Iterable)
[0ms] - boolean com.atlassian.stash.license.LicenseService.canLogin(Principal)
[0ms] - boolean com.atlassian.stash.user.PermissionService.hasGlobalPermission(StashUser,Permission)
[0ms] - Page com.atlassian.stash.user.UserService.findGroupsByUser(String,PageRequest)
[0ms] - Page com.atlassian.stash.internal.user.EffectivePermissionDao.findByGroup(Iterable,PageRequest)
[0ms] - boolean com.atlassian.stash.user.PermissionService.isRepositoryAccessible(Repository)
[1390ms] - scm-cache: upload-pack-request
[0ms] - Object com.atlassian.stash.user.SecurityService.doWithPermission(String,Permission,Operation)
[0ms] - Repository com.atlassian.stash.repository.RepositoryService.getBySlug(String,String)
[0ms] - InternalRepository com.atlassian.stash.internal.repository.RepositoryDao.getBySlug(String,String)
[0ms] - boolean com.atlassian.stash.user.PermissionService.isRepositoryAccessible(Repository)
[0ms] - Ticket com.atlassian.stash.throttle.ThrottleService.acquireTicket(String)
[62ms] - C:\Program Files (x86)\Git\bin\git.exe http-backend

T. Alexander Lystad added a comment - 24/Jan/2014 9:19 PM Ah, so I guess this is what is screwing us. 2014-01-24 11:11:51,203 | http-apr-7990-exec-22 | 671x952899x4 | TeamCityBuildServer | 1xrzail [10156ms] - "GET /scm/VP/vismapayroll.git/info/refs HTTP/1.1" [0ms] - com.atlassian.stash.internal.spring.security.StashAuthenticationFilter.requiresAuthentication [109ms] - Authentication org.springframework.security.authentication.AuthenticationProvider.authenticate(Authentication) [16ms] - attemptAuthentication - com.atlassian.stash.stash-auth-crowd-sso:crowdSsoAuthHandler [0ms] - CaptchaTicket com.atlassian.stash.internal.user.CaptchaService.checkCaptcha(String,CaptchaResponse) [0ms] - StashUser com.atlassian.stash.internal.user.CaptchaService.authenticateWithCaptcha(CaptchaTicket,UncheckedOperation) [93ms] - attemptAuthentication - com.atlassian.stash.stash-authentication:crowdHttpAuthHandler [93ms] - StashUser com.atlassian.stash.internal.user.CaptchaService.authenticateWithCaptcha(CaptchaTicket,UncheckedOperation) [93ms] - StashUser com.atlassian.stash.user.UserService.authenticate(String,String) [0ms] - InternalStashUser com.atlassian.stash.internal.user.StashUserDao.loadUser(User) [0ms] - boolean com.atlassian.stash.internal.user.InternalPermissionService.hasGlobalPermission(StashUserAuthenticationToken,Permission) [0ms] - Page com.atlassian.stash.user.UserService.findGroupsByUser(String,PageRequest) [0ms] - Page com.atlassian.stash.internal.user.EffectivePermissionDao.findByGroup(Iterable,PageRequest) [0ms] - Object com.atlassian.stash.user.SecurityService.doWithPermission(String,Permission,Operation) [0ms] - Repository com.atlassian.stash.repository.RepositoryService.getBySlug(String,String) [0ms] - InternalRepository com.atlassian.stash.internal.repository.RepositoryDao.getBySlug(String,String) [0ms] - boolean com.atlassian.stash.user.PermissionService.isRepositoryAccessible(Repository) [8641ms] - boolean com.atlassian.stash.scm.ScmRequestCheckService.checkActionAllowed(ScmRequest) [8625ms] - KeyedMessage com.atlassian.stash.license.LicenseService.getStatus() [8625ms] - int com.atlassian.stash.internal.user.InternalPermissionService.getCountOfUsersWithPermission(Permission) [0ms] - Page com.atlassian.stash.internal.user.EffectivePermissionDao.findUsers(Permission,PageRequest,Predicate) [0ms] - Page com.atlassian.stash.internal.user.EffectivePermissionDao.findGroups(Permission,PageRequest) [0ms] - Page com.atlassian.stash.user.UserService.findUsersByGroup(String,PageRequest) [0ms] - Set com.atlassian.stash.internal.user.StashUserDao.loadUsers(Iterable) [8610ms] - Page com.atlassian.stash.user.UserService.findUsersByGroup(String,PageRequest) [0ms] - Set com.atlassian.stash.internal.user.StashUserDao.loadUsers(Iterable) [15ms] - Page com.atlassian.stash.user.UserService.findUsersByGroup(String,PageRequest) [0ms] - Set com.atlassian.stash.internal.user.StashUserDao.loadUsers(Iterable) [0ms] - boolean com.atlassian.stash.license.LicenseService.canLogin(Principal) [0ms] - boolean com.atlassian.stash.user.PermissionService.hasGlobalPermission(StashUser,Permission) [0ms] - Page com.atlassian.stash.user.UserService.findGroupsByUser(String,PageRequest) [0ms] - Page com.atlassian.stash.internal.user.EffectivePermissionDao.findByGroup(Iterable,PageRequest) [0ms] - boolean com.atlassian.stash.user.PermissionService.isRepositoryAccessible(Repository) [1390ms] - scm-cache: upload-pack-request [0ms] - Object com.atlassian.stash.user.SecurityService.doWithPermission(String,Permission,Operation) [0ms] - Repository com.atlassian.stash.repository.RepositoryService.getBySlug(String,String) [0ms] - InternalRepository com.atlassian.stash.internal.repository.RepositoryDao.getBySlug(String,String) [0ms] - boolean com.atlassian.stash.user.PermissionService.isRepositoryAccessible(Repository) [0ms] - Ticket com.atlassian.stash.throttle.ThrottleService.acquireTicket(String) [62ms] - C:\Program Files (x86)\Git\bin\git.exe http-backend

Bitbucket Data Center

Licensed user count cache should be repopulated in the background to avoid blocking request threads

Workaround

[BSERV-4228] Licensed user count cache should be repopulated in the background to avoid blocking request threads

Bitbucket Data Center

Details

Description

Workaround

Attachments

Issue Links

Forms

Activity

[BSERV-4228] Licensed user count cache should be repopulated in the background to avoid blocking request threads

Collapse comment: T. Alexander Lystad added a comment - 20/Feb/2014 7:54 AM

Expand comment: T. Alexander Lystad added a comment - 20/Feb/2014 7:54 AM

Collapse comment: jhinch (Atlassian) added a comment - 20/Feb/2014 12:52 AM

Expand comment: jhinch (Atlassian) added a comment - 20/Feb/2014 12:52 AM

Collapse comment: T. Alexander Lystad added a comment - 20/Feb/2014 12:16 AM, Edited by T. Alexander Lystad - 20/Feb/2014 12:24 AM

Expand comment: T. Alexander Lystad added a comment - 20/Feb/2014 12:16 AM, Edited by T. Alexander Lystad - 20/Feb/2014 12:24 AM

Collapse comment: jhinch (Atlassian) added a comment - 20/Feb/2014 12:05 AM

Expand comment: jhinch (Atlassian) added a comment - 20/Feb/2014 12:05 AM

Collapse comment: T. Alexander Lystad added a comment - 24/Jan/2014 9:20 PM

Expand comment: T. Alexander Lystad added a comment - 24/Jan/2014 9:20 PM

Collapse comment: T. Alexander Lystad added a comment - 24/Jan/2014 9:19 PM

Expand comment: T. Alexander Lystad added a comment - 24/Jan/2014 9:19 PM

People

Dates

Backbone Issue Sync