Loading...

Type: Bug
Resolution: Fixed
Priority: High
Fix Version/s: 8.5.2, 8.6.0, 7.13.12
Affects Version/s: 8.0.0, 7.13.6, 8.3.3
Component/s: User Management - Groups
Labels:
None

Fixed in Long Term Support Release/s:

Download 7.13, 8.5
Introduced in Version:
7.13
Support reference count:
2
Symptom Severity:
Severity 2 - Major
Bug Fix Policy:
View Atlassian Server bug fix policy

Issue Summary

Having duplicated names for the group while having a large number of groups in Jira instance will cause DB "duplicate key value violates unique constraint" error. This will cause the error while loading the data for readUsersInGroupsByDirectory method. Which will lead to different side-effects and possible instance slowness.

Steps to Reproduce

Create more than:
- 30 000 memberships for postgresql database,
- 2 000 MSSQL database

Expected Results

Jira works healthy without downtimes

Actual Results

A lot of threads are waiting on the activeUsersCountForLicense cache in the DefaultApplicationRoleManager class
For instance you can find such threads:

parking to wait for  <0x0000000223ace570> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
	at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
	at net.sf.ehcache.concurrent.ReadWriteLockSync.lock(ReadWriteLockSync.java:50)
	at net.sf.ehcache.constructs.blocking.BlockingCache.acquiredLockForKey(BlockingCache.java:196)
	at net.sf.ehcache.constructs.blocking.BlockingCache.get(BlockingCache.java:158)
	at com.atlassian.cache.ehcache.LoadingCache.get(LoadingCache.java:79)
	at com.atlassian.cache.ehcache.DelegatingCache.get(DelegatingCache.java:108)
	at com.atlassian.jira.cache.DeferredReplicationCache.get(DeferredReplicationCache.java:48)
	at com.atlassian.jira.application.DefaultApplicationRoleManager.lambda$getUserCountCurrentValue$16(DefaultApplicationRoleManager.java:583)
	at com.atlassian.jira.application.DefaultApplicationRoleManager$$Lambda$915/480723923.apply(Unknown Source)
	at com.atlassian.fugue.Option$Some.fold(Option.java:421)
	at com.atlassian.jira.application.DefaultApplicationRoleManager.getUserCountCurrentValue(DefaultApplicationRoleManager.java:581)
	at com.atlassian.jira.application.DefaultApplicationRoleManager.getUserCount(DefaultApplicationRoleManager.java:574)
	at com.spartez.pokerng.adapters.Jira70ApplicationRoleManager.getJiraSoftwareUserCount(Jira70ApplicationRoleManager.java:10)
	at com.spartez.pokerng.data.AnalyticsStatusDataProvider.guessTier(AnalyticsStatusDataProvider.java:78)
	at com.spartez.pokerng.data.AnalyticsStatusDataProvider.provideData(AnalyticsStatusDataProvider.java:55)
	at com.spartez.pokerng.data.SafeDataProvider$1.write(SafeDataProvider.java:29)
...

and you can find in the log stack traces which show that we failed to refresh activeUsersCountForLicense cache for instance:

Caused by: com.atlassian.jira.exception.DataAccessException: org.ofbiz.core.entity.GenericEntityException: Batch entry 16,143 insert into temp8 (item) values ('atlassian sysadmin') was aborted: ERROR: duplicate key value violates unique constraint "temp8_pkey"      Detail: Key (item)=(atlassian sysadmin) already exists.  Call getNextException to see other errors in the batch. (Batch entry 16,143 insert into temp8 (item) values ('atlassian sysadmin') was aborted: ERROR: duplicate key value violates unique constraint "temp8_pkey"      Detail: Key (item)=(atlassian sysadmin) already exists.  Call getNextException to see other errors in the batch.)
	at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.findListIteratorByCondition(DefaultOfBizDelegator.java:401) [classes/:?]
	at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.findListIteratorByCondition(WrappingOfBizDelegator.java:278) [classes/:?]
	at com.atlassian.jira.crowd.embedded.ofbiz.OfBizInternalMembershipDao.findChildrenOfGroupsGeneric(OfBizInternalMembershipDao.java:306) [classes/:?]
	at com.atlassian.jira.crowd.embedded.ofbiz.OfBizInternalMembershipDao.findUserChildrenOfGroups(OfBizInternalMembershipDao.java:289) [classes/:?]
	at com.atlassian.jira.crowd.embedded.ofbiz.OfBizInternalMembershipDao.findUserChildrenOfGroups(OfBizInternalMembershipDao.java:58) [classes/:?]
	at com.atlassian.jira.application.DefaultApplicationRoleManager.readUsersInGroupsByDirectory(DefaultApplicationRoleManager.java:1035) [classes/:?]
	at com.atlassian.jira.application.DefaultApplicationRoleManager.readUsersInGroupsExcluding(DefaultApplicationRoleManager.java:1066) [classes/:?]
	at com.atlassian.jira.application.DefaultApplicationRoleManager.readUsersInGroups(DefaultApplicationRoleManager.java:1048) [classes/:?]
	at com.atlassian.jira.application.DefaultApplicationRoleManager.countUsersInGroups(DefaultApplicationRoleManager.java:1100) [classes/:?]
	at com.atlassian.jira.application.DefaultApplicationRoleManager.countApplicationRoleUsers(DefaultApplicationRoleManager.java:955) [classes/:?]
	at com.atlassian.jira.application.DefaultApplicationRoleManager.countApplicationUsers(DefaultApplicationRoleManager.java:938) [classes/:?]
	at com.atlassian.fugue.Option.map(Option.java:272) [fugue-2.6.0.jar:?]
	at com.atlassian.jira.application.DefaultApplicationRoleManager.lambda$new$1(DefaultApplicationRoleManager.java:197) [classes/:?]
	at com.atlassian.cache.ehcache.wrapper.ValueProcessorAtlassianCacheLoaderDecorator.load(ValueProcessorAtlassianCacheLoaderDecorator.java:26) [atlassian-cache-ehcache-3.2.0.jar:?]
	at com.atlassian.cache.ehcache.LoadingCache.getFromLoader(LoadingCache.java:133) [atlassian-cache-ehcache-3.2.0.jar:?]
	at com.atlassian.cache.ehcache.SynchronizedLoadingCacheDecorator.synchronizedLoad(SynchronizedLoadingCacheDecorator.java:29) [atlassian-cache-ehcache-3.2.0.jar:?]
	at com.atlassian.cache.ehcache.LoadingCache.loadValueAndReleaseLock(LoadingCache.java:101) [atlassian-cache-ehcache-3.2.0.jar:?]
	at com.atlassian.cache.ehcache.LoadingCache.get(LoadingCache.java:80) [atlassian-cache-ehcache-3.2.0.jar:?]
	at com.atlassian.cache.ehcache.DelegatingCache.get(DelegatingCache.java:108) [atlassian-cache-ehcache-3.2.0.jar:?]	... 300 more

You may also see 500 access issues when loading the Manage Apps page or Versions and Licenses:

Workaround

1. Decrease number of groups to less than 30 000. It can be checked how many records is in database by this sql:

SELECT count (*),lower_child_name
FROM cwd_membership
WHERE directory_id=DIRECTORYID
AND membership_type='GROUP_GROUP'
AND lower_parent_name in (
	SELECT lower_group_name
	FROM cwd_group
	WHERE directory_id=DIRECTORYID
)

2. Filter out unneeded groups/users from LDAP by editing LDAP filters.
or

3. Make sure that you do not have duplication in the lower names of the group and nested groups.

is related to

JRASERVER-64384 ActiveUsersCountForLicense cache is populated inefficiently leading to performance degradation across the application

Closed

links to

https://bulldog.internal.atlassian.com/browse/PSR-359

mentioned in: Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...

relates to: PSR-359 Loading...

(6 mentioned in, 1 relates to)

Details

Description

Issue Summary

Steps to Reproduce

Expected Results

Actual Results

Workaround

Attachments

Issue Links

Forms

Activity

People

Dates