-
Bug
-
Resolution: Fixed
-
Medium
-
2.7.1
Summary
When there are multiple directories and membership aggregation is disabled for an applicatoin, group membership queries require an isCanonical check on each user. In order to do this isCanonical finds the user by name in order to shallow the user. This results in n+1 queries to the database. This can cause massive performance issues when the host application (in this case Stash) when calculating licensed user counts.
Steps to Reproduce
It is not entirely clear at this point how to accurately replicate the problem. It involves configuring nested groups and multiple user directories, then using the application such that it retrieves user groups.
Expected Results
Crowd retrieves nested groups with minimal performance impact.
Actual Results
There is a notable performance impact on the system and it presents with symptoms of performance problems.
Verification
Check for the presence of multiple threads utilising com.atlassian.crowd.manager.application.ApplicationServiceGeneric.isCanonical and/or com.atlassian.crowd.manager.application.ApplicationServiceGeneric.searchNestedGroupRelationships. For example from JIRA:
"http-bio-8080-exec-12889" daemon prio=10 tid=0x00007f9e7cb26800 nid=0xa4a runnable [0x00007f9e3125e000] java.lang.Thread.State: RUNNABLE at java.util.HashMap$HashIterator.nextEntry(Unknown Source) at java.util.HashMap$KeyIterator.next(Unknown Source) at java.util.AbstractCollection.addAll(Unknown Source) at com.atlassian.crowd.model.application.DirectoryMapping.<init>(DirectoryMapping.java:70) at com.atlassian.jira.crowd.embedded.ofbiz.OfBizApplication$1.apply(OfBizApplication.java:71) at com.atlassian.jira.crowd.embedded.ofbiz.OfBizApplication$1.apply(OfBizApplication.java:68) at com.google.common.collect.Lists$TransformingRandomAccessList.get(Lists.java:451) at java.util.AbstractList$Itr.next(Unknown Source) at com.google.common.collect.Iterators$8.next(Iterators.java:781) at com.google.common.collect.Iterators$7.computeNext(Iterators.java:644) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) at com.google.common.collect.Iterators.getNext(Iterators.java:860) at com.google.common.collect.Iterables.getFirst(Iterables.java:759) at com.atlassian.crowd.manager.application.ApplicationServiceGeneric.isCanonical(ApplicationServiceGeneric.java:1638) at com.atlassian.crowd.manager.application.ApplicationServiceGeneric$1.apply(ApplicationServiceGeneric.java:1587) at com.google.common.collect.Iterators$7.computeNext(Iterators.java:645) at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:141) at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:136) at com.google.common.collect.Iterables$10$1.hasNext(Iterables.java:884) at com.google.common.collect.Lists.newArrayList(Lists.java:138) at com.google.common.collect.Lists.newArrayList(Lists.java:119) at com.atlassian.crowd.manager.application.AggregatorImpl.constrainResults(ResultsAggregator.java:174) at com.atlassian.crowd.manager.application.ApplicationServiceGeneric.searchNestedGroupRelationships(ApplicationServiceGeneric.java:1582) at com.atlassian.crowd.embedded.core.CrowdServiceImpl.searchNestedGroupRelationships(CrowdServiceImpl.java:238) at com.atlassian.crowd.embedded.core.CrowdServiceImpl.search(CrowdServiceImpl.java:158) at com.atlassian.crowd.embedded.core.DelegatingCrowdService.search(DelegatingCrowdService.java:60) at com.atlassian.crowd.embedded.core.FilteredCrowdServiceImpl.search(FilteredCrowdServiceImpl.java:143) at com.atlassian.jira.security.groups.DefaultGroupManager.getUsersInGroup(DefaultGroupManager.java:107) at com.atlassian.jira.security.roles.actor.GroupRoleActorFactory$GroupRoleActor.getUsers(GroupRoleActorFactory.java:79) at com.atlassian.jira.security.roles.DefaultRoleActorsImpl.getUsers(DefaultRoleActorsImpl.java:51) at com.atlassian.jira.security.roles.CachingProjectRoleAndActorStore$CachedRoleActors.getUsers(CachingProjectRoleAndActorStore.java:415) at com.atlassian.jira.notification.type.ProjectRoleSecurityAndNotificationType.getUsersFromRole(ProjectRoleSecurityAndNotificationType.java:235) at com.atlassian.jira.notification.type.ProjectRoleSecurityAndNotificationType.getUsers(ProjectRoleSecurityAndNotificationType.java:138) at com.atlassian.jira.scheme.AbstractSchemeManager.getUsers(AbstractSchemeManager.java:819) at com.atlassian.jira.permission.WorkflowBasedPermissionSchemeManager.getUsers(WorkflowBasedPermissionSchemeManager.java:73) at com.atlassian.jira.bc.user.search.DefaultAssigneeService$AssignableUsers.findAllAsSet(DefaultAssigneeService.java:428) at com.atlassian.jira.bc.user.search.DefaultAssigneeService$AssignableUsers.findAll(DefaultAssigneeService.java:423) at com.atlassian.jira.bc.user.search.DefaultAssigneeService.getSuggestedAssignees(DefaultAssigneeService.java:70) at com.atlassian.jira.issue.fields.Assignees.optionsForFrotherControl(Assignees.java:70) at com.atlassian.jira.issue.fields.AssigneeSystemField.getEditHtml(AssigneeSystemField.java:168) at com.atlassian.jira.issue.fields.screen.AbstractFieldScreenLayoutItem.getEditHtml(AbstractFieldScreenLayoutItem.java:78) at com.atlassian.jira.issue.fields.screen.FieldScreenRenderLayoutItemImpl.getEditHtml(FieldScreenRenderLayoutItemImpl.java:62) at com.atlassian.jira.issue.fields.rest.FieldHtmlFactoryImpl.getEditFields(FieldHtmlFactoryImpl.java:151) at sun.reflect.GeneratedMethodAccessor1026.invoke(Unknown Source)
Notes
In JIRA, this can also cause performance problems. For example, some user actions in JIRA like com.atlassian.jira.components.issueeditor.action.AjaxIssueEditAction.doDefault and com.atlassian.jira.issue.fields.screen.FieldScreenRenderLayoutItemImpl.getEditHtml, there are many) require JIRA to calculate the permissions, which involves calling com.atlassian.jira.security.groups.DefaultGroupManager.getUsersInGroup. JIRA delegates this operation to ApplicationServiceGeneric.searchNestedGroupRelationships in its embedded Crowd. Embedded Crowd is dependent on these classes from Crowd. This can be an expensive calculation which increases with the number of users. Unfortunately the loop inside that method causes many queries to the database, many of them to fetch the directory mappings which are not cached.
- is duplicated by
-
CWD-4689 Improve isCanonical check in InMemoryNonAggregatingSearchStrategy
- Closed
- is incorporated by
-
-
- Closed
-
- is related to
-
-
- Closed
-
- was split from
-
-
- Closed
-
![[Extranet] Page](/images/icons/generic_link_16.png "[Extranet] Page"){kind=icon}
[CWD-3809] ApplicationServiceGeneric#searchNestedGroupRelationships performs n+1 queries in order to shadow users
Manuel Arranz, what I can suggest from our own experience was to first double check to ensure even the local directories in your apps (JIRA, etc.) are set to not use nested groups, as well as in the Crowd connectors in the apps, and in the Crowd Directories.
Once verified all off, our performance fix was to upgrade JIRA to 6.3.9 (or above, some performance fixes in 6.3.8/9 that were important for this), Crowd to 2.8 or above, and lastly (and most importantly) to upgrade to Java JDK8 on JIRA. We were using JDK7 before and that change fixed our performance issues once and for all. May not apply to your case, but hopefully it helps. We've gone 10 months now without any sort of performance trouble in JIRA under this setup.
Cheers,
Mike
I think our problem is similar to the one reported by Mike Rocheleau. Our JIRA is connected to Crowd and we have multiple Directories allowed but we don't use nested groups. Is there another ticket where this problem is being tracked?
Best Regards
Hi, in our JIRA instance we are seeing high cpu consumption in this method "com.atlassian.crowd.model.application.DirectoryMapping.<init>” and suspect it could be related to this issue and the fact of Directory Mappings not being cached.
Any news about this issue?
Hi mike.rocheleau. This will not affect JIRA talking to Crowd, even if the Remote Crowd directory in JIRA is uncached. The directory aggregation happens in Crowd.
However, it looks like you're hitting a performance problem that should be investigated. I suggest you open a support ticket if you haven't done that yet, and provide as much information as you can so we can reproduce and investigate the problem. Thank you.
Would this apply to JIRA, talking to Crowd, where we have multiple Directories allowed in Crowd, for the JIRA Application?
We have no nested group searches enabled either in JIRA or Crowd but are seeing spikes where massive amount of 'NestedGroupRelationships' threads are performed, almost halting service in JIRA.
In Crowd 2.8, the new directory aggregation option (
CWD-3870) will avoid checking for shadowing if enabled. I've adjusted the description appropriately. For cases where the aggregating membership model is appropriate or where you can rearrange your directories so it becomes appropriate (see https://confluence.atlassian.com/display/CROWD/Effective+memberships+with+multiple+directories), you should see a significant improvement for these queries.