-
Bug
-
Resolution: Fixed
-
Highest
-
7.11.1
-
3
-
Severity 2 - Major
-
41
-
Issue Summary
Confluence times out when performing permission checks for a page with lots of attachments macros.
Improve the performance of searchGroupedByName method.
Steps to Reproduce
- Add a page with several attachment macros referencing attachments on current page and other pages.
- Publish page
- View page.
Expected Results
The page loads within a resemble time.
Actual Results
Page eventually times out with error.
The below thread dump captures the time spent loading the page:
"http-nio-8090-exec-2" #226 daemon prio=5 os_prio=0 cpu=1205421.88ms elapsed=180041.76s tid=0x000000003e65c000 nid=0xb90 runnable [0x00000000560ce000] java.lang.Thread.State: RUNNABLE at org.hibernate.type.AbstractStandardBasicType.isEqual(AbstractStandardBasicType.java:198) at org.hibernate.type.AbstractStandardBasicType.isSame(AbstractStandardBasicType.java:186) ... at com.atlassian.crowd.embedded.hibernate2.GroupingHibernateSearch.doInHibernate(GroupingHibernateSearch.java:61) ... at com.atlassian.crowd.directory.AbstractInternalDirectory.searchGroupRelationshipsGroupedByName(AbstractInternalDirectory.java:890) at com.atlassian.crowd.directory.DbCachingRemoteDirectory.searchGroupRelationshipsGroupedByName(DbCachingRemoteDirectory.java:1154) at com.atlassian.crowd.manager.directory.RemoteDirectorySearcher.searchDirectGroupRelationshipsGroupedByName(RemoteDirectorySearcher.java:97) at com.atlassian.crowd.manager.directory.RemoteDirectorySearcher.lambda$nestedGroupsProvider$3(RemoteDirectorySearcher.java:220) at com.atlassian.crowd.manager.directory.RemoteDirectorySearcher$$Lambda$1764/0x000000080240a840.getDirectlyRelatedGroups(Unknown Source) at com.atlassian.crowd.manager.directory.nestedgroups.NestedGroupsProviderImpl.getDirectlyRelatedGroups(NestedGroupsProviderImpl.java:26) at com.atlassian.crowd.manager.directory.nestedgroups.NestedGroupsIterator.fetchRelatedGroups(NestedGroupsIterator.java:40) at com.atlassian.crowd.manager.directory.nestedgroups.NestedGroupsIterator.fetchNextIfNeeded(NestedGroupsIterator.java:65) at com.atlassian.crowd.manager.directory.nestedgroups.NestedGroupsIterator.hasNext(NestedGroupsIterator.java:50) at com.atlassian.crowd.manager.directory.nestedgroups.NestedGroupsIterator.anyMatch(NestedGroupsIterator.java:82) at com.atlassian.crowd.manager.directory.RemoteDirectorySearcher.isUserNestedGroupMember(RemoteDirectorySearcher.java:135) at com.atlassian.crowd.manager.directory.DirectoryManagerGeneric.isUserNestedGroupMember(DirectoryManagerGeneric.java:807) at com.atlassian.crowd.manager.application.ApplicationServiceGeneric$3.fallibleCheckForEntity(ApplicationServiceGeneric.java:1050) at com.atlassian.crowd.manager.application.ApplicationServiceGeneric$DirectoryPredicate.apply(ApplicationServiceGeneric.java:1015) at com.atlassian.crowd.manager.application.ApplicationServiceGeneric$DirectoryPredicate.apply(ApplicationServiceGeneric.java:1011) ... at com.atlassian.crowd.manager.application.ApplicationServiceGeneric.isUserNestedGroupMember(ApplicationServiceGeneric.java:1528) at com.atlassian.crowd.embedded.core.CrowdServiceImpl.isUserMemberOfGroup(CrowdServiceImpl.java:230) ... at com.atlassian.confluence.security.ContentPermission.isPermitted(ContentPermission.java:150) at com.atlassian.confluence.security.ContentPermissionSet.isPermitted(ContentPermissionSet.java:76) at com.atlassian.confluence.core.DefaultContentPermissionManager.hasInheritedContentLevelViewPermissions(DefaultContentPermissionManager.java:247) at com.atlassian.confluence.core.DefaultContentPermissionManager.hasContentLevelPermission(DefaultContentPermissionManager.java:174) ... at com.atlassian.confluence.security.delegate.PagePermissionsDelegate.hasContentLevelViewPermission(PagePermissionsDelegate.java:123) at com.atlassian.confluence.security.delegate.PagePermissionsDelegate.canView(PagePermissionsDelegate.java:20) at com.atlassian.confluence.security.delegate.PagePermissionsDelegate.canView(PagePermissionsDelegate.java:13) at com.atlassian.confluence.security.delegate.TargetToLatestVersionDecorator.canView(TargetToLatestVersionDecorator.java:24) at com.atlassian.confluence.security.delegate.SharedAccessInterceptor$$Lambda$3158/0x000000080309f040.test(Unknown Source) at com.atlassian.confluence.security.delegate.SharedAccessInterceptor.checkAccess(SharedAccessInterceptor.java:87) at com.atlassian.confluence.security.delegate.SharedAccessInterceptor.canView(SharedAccessInterceptor.java:25) at com.atlassian.confluence.security.Permission$1.checkAgainst(Permission.java:17) at com.atlassian.confluence.security.DefaultPermissionManager.hasPermissionNoExemptions(DefaultPermissionManager.java:100) at com.atlassian.confluence.security.delegate.AttachmentPermissionsDelegate.canCreate(AttachmentPermissionsDelegate.java:120) at com.atlassian.confluence.security.DefaultPermissionManager.hasCreatePermission(DefaultPermissionManager.java:115) ...
Notes
Fastpermissions didn't help in this instance.
Workaround
Currently there is no known workaround for this behavior. A workaround will be added here when available
- is related to
-
-
- Gathering Impact
-
[CONFSERVER-63527] Viewing a large page with attachments macro causes slow loads and timeouts.
A fix for this issue is available to Server and Data Center customers in Confluence 7.12.1
Upgrade now or check out the Release Notes to see what other issues are resolved.
A fix for this issue is available to Server and Data Center customers in Confluence 7.11.3
Upgrade now or check out the Release Notes to see what other issues are resolved.
This bug brought a lot of trouble to us.
We are running a 7.11.1 in production without a sync to our active directory, because of the bug https://jira.atlassian.com/browse/CONFSERVER-61194. We needed to restart the app-server while a sync job to the ad. So the sync gets stuck and there is no way get it working again.
So we need to upgrade to version 7.12 and hoped for a fix. On the other hand we need to active the "nested" group function for a rollout of our installation.
So we updated to 7.12 and activated the "nested" group-feature. The update failed 2x times because if some load goes to our installation the database cpu utilisation got up to 100% and caused a outage of the whole production system.
It is a very frustrating situation for us and we need a fix to go on.
We are having problems with simply viewing restricted pages rather than attachments.
All sub-pages of a page with complex permissions applied take a long time to display the page.
2021-04-15 16:23:44,905 DEBUG [https-jsse-nio-443-exec-19] [atlassian.util.profiling.Timers] onClose [7245.6ms] - /display/~finix/test555
[193.4ms] - PermissionManager.isConfluenceAdministrator()
[192.2ms] [count: 6, avg: 32.0ms] - CrowdService.isUserMemberOfGroup()
[189.2ms] [count: 7, avg: 27.0ms] - MembershipDao.searchGroupedByName()
[5.2ms] - UserAccessor.getExistingUserByKey()
[6737.8ms] - XW Interceptor: Before defaultStack: /pages/viewpage.action (ViewPageAction.execute())
[6731.5ms] - SpaceAwareInterceptor.intercept()
[6731.5ms] - PageAwareInterceptor.intercept()
[11.3ms] - PageManager.getPageWithComments()
[18.6ms] - PermissionManager.hasPermission()
[18.5ms] - DefaultSpacePermissionManager.hasPermissionNoExemptions(VIEWSPACE, finix, ~finix)
[18.4ms] [count: 2, avg: 9.2ms] - CrowdService.isUserMemberOfGroup()
[18.0ms] [count: 3, avg: 6.0ms] - MembershipDao.searchGroupedByName()
[491.0ms] [count: 3, avg: 163.7ms] - PermissionManager.hasPermission()
[489.6ms] [count: 8, avg: 61.2ms] - CrowdService.isUserMemberOfGroup()
[484.0ms] [count: 13, avg: 37.2ms] - MembershipDao.searchGroupedByName()
[6184.2ms] - XW Interceptor: After defaultStack: /pages/viewpage.action (ViewPageAction.execute())
[6184.2ms] - XW Interceptor: After validatingStack: /pages/viewpage.action (ViewPageAction.execute())
[6179.4ms] - XW View: doExecute(/pages/viewpage.vm)
[6177.3ms] - ApplyDecoratorDirective.render()
[6175.9ms] - Rendering velocity template(/decorators/root.vmd)
[6175.7ms] - ApplyDecoratorDirective.render()
[6175.2ms] - Rendering velocity template(/decorators/page.vmd)
[113.5ms] - ContentEntityManager.getContributionStatusByUser()
[208.0ms] - PermissionManager.hasPermission()
[207.8ms] [count: 6, avg: 34.6ms] - CrowdService.isUserMemberOfGroup()
[205.7ms] [count: 11, avg: 18.7ms] - MembershipDao.searchGroupedByName()
[99.2ms] - CrowdService.search()
[8.9ms] - MembershipDao.search()
[41.3ms] [count: 6, avg: 6.9ms] - MembershipDao.searchGroupedByName()
[7.1ms] - MembershipDao.search()
[40.9ms] [count: 6, avg: 6.8ms] - MembershipDao.searchGroupedByName()
[4472.1ms] - Parse: /decorators/includes/content-navigation.vm
[224.7ms] - PermissionManager.hasPermissionNoExemptions()
[45.2ms] - DefaultSpacePermissionManager.hasPermissionNoExemptions(EDITSPACE, finix, ~finix)
[45.1ms] [count: 2, avg: 22.6ms] - CrowdService.isUserMemberOfGroup()
[44.6ms] [count: 3, avg: 14.9ms] - MembershipDao.searchGroupedByName()
[179.3ms] [count: 6, avg: 29.9ms] - CrowdService.isUserMemberOfGroup()
[177.4ms] [count: 11, avg: 16.1ms] - MembershipDao.searchGroupedByName()
[596.3ms] [count: 3, avg: 198.8ms] - PermissionManager.hasPermission()
[595.8ms] [count: 8, avg: 74.5ms] - CrowdService.isUserMemberOfGroup()
[590.5ms] [count: 13, avg: 45.4ms] - MembershipDao.searchGroupedByName()
[139.3ms] - PermissionManager.hasPermissionNoExemptions()
[139.2ms] [count: 6, avg: 23.2ms] - CrowdService.isUserMemberOfGroup()
[137.7ms] [count: 11, avg: 12.5ms] - MembershipDao.searchGroupedByName()
[648.5ms] [count: 3, avg: 216.2ms] - PermissionManager.hasPermission()
[648.1ms] [count: 8, avg: 81.0ms] - CrowdService.isUserMemberOfGroup()
[643.1ms] [count: 13, avg: 49.5ms] - MembershipDao.searchGroupedByName()
[83.5ms] - CrowdService.search()
[7.6ms] - MembershipDao.search()
[35.5ms] [count: 6, avg: 5.9ms] - MembershipDao.searchGroupedByName()
[6.0ms] - MembershipDao.search()
[33.2ms] [count: 6, avg: 5.5ms] - MembershipDao.searchGroupedByName()
[1525.9ms] [count: 8, avg: 190.7ms] - PermissionManager.hasPermission()
[1524.7ms] [count: 13, avg: 117.3ms] - CrowdService.isUserMemberOfGroup()
[1510.0ms] [count: 18, avg: 83.9ms] - MembershipDao.searchGroupedByName()
[196.9ms] - CrowdService.search()
[6.4ms] - MembershipDao.search()
[50.8ms] [count: 6, avg: 8.5ms] - MembershipDao.searchGroupedByName()
[7.1ms] - MembershipDao.search()
[131.2ms] [count: 6, avg: 21.9ms] - MembershipDao.searchGroupedByName()
[208.3ms] - PermissionManager.hasPermission()
[208.1ms] [count: 6, avg: 34.7ms] - CrowdService.isUserMemberOfGroup()
[206.3ms] [count: 11, avg: 18.8ms] - MembershipDao.searchGroupedByName()
[163.4ms] - PermissionManager.hasPermissionNoExemptions()
[163.2ms] [count: 6, avg: 27.2ms] - CrowdService.isUserMemberOfGroup()
[161.5ms] [count: 11, avg: 14.7ms] - MembershipDao.searchGroupedByName()
[635.8ms] [count: 3, avg: 211.9ms] - PermissionManager.hasPermission()
[635.1ms] [count: 8, avg: 79.4ms] - CrowdService.isUserMemberOfGroup()
[629.8ms] [count: 13, avg: 48.4ms] - MembershipDao.searchGroupedByName()
[172.3ms] - PermissionManager.hasPermission()
[172.2ms] [count: 6, avg: 28.7ms] - CrowdService.isUserMemberOfGroup()
[170.3ms] [count: 11, avg: 15.5ms] - MembershipDao.searchGroupedByName()
[4.4ms] - DefaultRenderer.render
[4.3ms] - DefaultRenderer.renderWithResult
[458.4ms] - Parse: /decorators/includes/page-content-includes.vm
[457.3ms] - Parse: /template/includes/page-move-templates.vm
[286.9ms] - PermissionManager.hasPermission()
[286.7ms] [count: 6, avg: 47.8ms] - CrowdService.isUserMemberOfGroup()
[284.9ms] [count: 11, avg: 25.9ms] - MembershipDao.searchGroupedByName()
[169.9ms] - PermissionManager.hasRemoveHierarchyPermission()
[169.7ms] [count: 6, avg: 28.3ms] - CrowdService.isUserMemberOfGroup()
[167.9ms] [count: 11, avg: 15.3ms] - MembershipDao.searchGroupedByName()
[2.1ms] - Parse: /decorators/includes/page-metadata.vm
[593.1ms] - ApplyDecoratorDirective.render()
[592.5ms] - Rendering velocity template(/decorators/root.vmd)
[592.2ms] - ApplyDecoratorDirective.render()
[591.7ms] - Rendering velocity template(/decorators/components/comments.vmd)
[169.0ms] - PermissionManager.hasCreatePermission()
[151.7ms] [count: 6, avg: 25.3ms] - CrowdService.isUserMemberOfGroup()
[149.9ms] [count: 11, avg: 13.6ms] - MembershipDao.searchGroupedByName()
[17.1ms] - DefaultSpacePermissionManager.hasPermissionNoExemptions(COMMENT, finix, ~finix)
[17.0ms] [count: 2, avg: 8.5ms] - CrowdService.isUserMemberOfGroup()
[16.7ms] [count: 3, avg: 5.6ms] - MembershipDao.searchGroupedByName()
[174.3ms] - Rendering velocity template(templates/quick-comment.vm)
[172.7ms] - PermissionManager.hasCreatePermission()
[172.6ms] [count: 6, avg: 28.8ms] - CrowdService.isUserMemberOfGroup()
[170.6ms] [count: 11, avg: 15.5ms] - MembershipDao.searchGroupedByName()
[246.1ms] - PermissionManager.hasCreatePermission()
[245.9ms] [count: 6, avg: 41.0ms] - CrowdService.isUserMemberOfGroup()
[244.1ms] [count: 11, avg: 22.2ms] - MembershipDao.searchGroupedByName()
[6.9ms] - Parse: /pages/page-breadcrumbs.vm
[6.1ms] - Parse: /breadcrumbs.vm
[12.7ms] - Parse: /template/includes/space-tools-web-items.vm
[12.4ms] - system.space.tools
[10.7ms] - PermissionManager.hasPermission()
[10.7ms] - DefaultSpacePermissionManager.hasPermissionNoExemptions(EXPORTSPACE, finix, ~finix)
[10.6ms] - CrowdService.isUserMemberOfGroup()
[10.4ms] [count: 2, avg: 5.2ms] - MembershipDao.searchGroupedByName()
[291.1ms] - SiteMesh: applyDecorator: main (/decorators/main.vmd)
[286.5ms] - Rendering velocity template(/decorators/main.vmd)
[103.5ms] - Parse: /decorators/includes/header.vm
[12.5ms] [count: 3, avg: 4.2ms] - ZduManager.getUpgradeStatus()
[69.7ms] - VelocityFriendlyPageBuilderService.getConfluenceResourceTags
[3.7ms] - ZduManager.getUpgradeStatus()
[18.2ms] [count: 5, avg: 3.6ms] - UserAccessor.getExistingUserByKey()
[177.6ms] - Parse: /decorators/includes/common-header.vm
[144.6ms] - Parse: /decorators/includes/aui-header-primary.vm
[120.4ms] - CrowdService.search()
[6.7ms] - MembershipDao.search()
[41.6ms] [count: 6, avg: 6.9ms] - MembershipDao.searchGroupedByName()
[5.8ms] - MembershipDao.search()
[65.1ms] [count: 6, avg: 10.9ms] - MembershipDao.searchGroupedByName()
[23.7ms] - Parse: /decorators/includes/aui-header-secondary.vm
[12.3ms] [count: 3, avg: 4.1ms] - SpaceManager.getPersonalSpace()
[8.1ms] - Rendering soy template(com.atlassian.plugins.atlassian-nav-links-plugin:rotp-menu, navlinks.templates.appswitcher.switcher)
[6.8ms] [count: 2, avg: 3.4ms] - UserAccessor.getExistingUserByKey()
@Lukas Reichart:
We have analyzed the cache with Atlassian Support and they said the cache is fine.
The problem seems to be due to a different method with permission checks that has been introduced with Confluence 7.11 (from what I understood).
If you add the user to the confluence-administrators group, the problem disappears for this user, because for the admins, these permission checks are skipped. So, adding the users to this group is the only workaround (apart from rebuilding all affected pages and make them less complex). Of course, giving users full admin permissions is not a workaround that can be used in most organizations...
On the confluence database I see a lot of queries of the format:
SELECT internalgr1_.group_name AS col_0_0_, hibernatem0_.child_group_id AS col_1_0_, internalgr2_.id AS id1_28_, internalgr2_.group_name AS group_na2_28_, internalgr2_.lower_group_name AS lower_gr3_28_, internalgr2_.active AS active4_28_, internalgr2_.local AS local5_28_, internalgr2_.created_date AS created_6_28_, internalgr2_.updated_date AS updated_7_28_, internalgr2_.description AS descript8_28_, internalgr2_.group_type AS group_ty9_28_, internalgr2_.directory_id AS directo10_28_, internalgr2_.external_id AS externa11_28_ FROM cwd_membership hibernatem0_ CROSS JOIN cwd_group internalgr1_ INNER JOIN cwd_group internalgr2_ ON hibernatem0_.child_group_id=internalgr2_.id WHERE hibernatem0_.parent_id=internalgr1_.id AND (hibernatem0_.child_user_id IS NULL) AND (internalgr1_.lower_group_name IN ($1, $2, $3, $4, $5, $6, $7, $8, $9, $10, $11, $12, $13, $14, $15, $16, $17, $18, $19, $20, $21, $22, $23, $24, $25, $26, $27, $28, $29, $30, $31, $32, $33, $34, $35, $36, $37, $38, ...
which for me would be another indicator that this is a caching-related problem?
My organization is also affected by this problem, pages with many attachments are essentially unusable. Here an extract from a profiling session:
[222.5ms] - PermissionManager.hasCreatePermission() [221.9ms] [count: 14, avg: 15.8ms] - CrowdService.isUserMemberOfGroup() [216.2ms] [count: 16, avg: 13.5ms] - MembershipDao.searchGroupedByName()
if you now have a more complex permission structure with many groups and many attachments and the permission check for each attachment is >200 ms you get page load times of around 20s - 30s.
Maybe the issue is related to the caching of group memberships? Because it also does not get better with repeated page loads.
Our organization is severely affected by this problem since the upgrade to Confluence 7.11.
We have hundreds of complex pages that can't be displayed any longer due to a timeout. Virtually leaves Confluence partially unusable for us.
Very annoying to say the least...
A fix for this issue is available to Server and Data Center customers in Confluence 7.13.0
Upgrade now or check out the Release Notes to see what other issues are resolved.