Loading...

XML

Word

Printable

Details

Type: Bug
Resolution: Fixed
Priority: Highest
Fix Version/s: 7.3.1
Affects Version/s: 6.0.4, 6.4.3, 6.6.3
Component/s: User - Global / Space Permissions
Labels:
- architectural
- l1l2
- p20
- pse-request
- scale-team
- tn-m

Support reference count:
94
Symptom Severity:
Severity 1 - Critical
UIS:
572
Bug Fix Policy:
View Atlassian Server bug fix policy

Description

Summary

In large environments, where spaces may contain several thousand pages, overall performance of Confluence can degrade due to required permissions lookups for pages in the page tree.

Actual Behavior

When a page tree is in use and a user views a page in a space using page tree, there is a permissions lookup which takes place for every page listed in the page tree. If the space has several thousand pages, this lookup can become very expensive when performed for multiple users accessing the space.

The performance impact tends to happen when there is a concurrent cache lock issue during the cache eviction. These permissions lookups can take a long time and when multiple users view pages in the space (causing this action to take place), it causes a new thread to be used for each user. If many or all threads are in use for this action due to the cache lock, it can lead to performance problems on the site as fewer threads are available for other operations.

Expected Behavior

Permissions lookups, regardless of whether they are cached, should take place in the background, be less expensive, and not affect the performance of the site.

This issue was resolved in Confluence 7.3. We now load a maximum of 200 pages at each level of the page hierarchy in the sidebar. A Show all pages button will appear if there are pages that have not been loaded. You can change this page tree behaviour using the following system properties:

page-tree.partial-loading-batch-size
page-tree.partial-loading.disable

See Recognized System Properties more information.

Symptoms

From the user perspective, the site may be sluggish and simply display a waiting (spinning) icon while the browser waits on the server to respond to a page view request.

This issue can be confirmed via thread dumps, and you may find hints in a heap dump too.

Thread Dump Examples

Thread dumps should be taken to confirm the issue. The key piece here is the blocking (RUNNABLE) thread and all blocked threads were executing this piece of code:

net.sf.hibernate.cache.ReadWriteCache.get(ReadWriteCache.java:69)

In this example case we see the following thread exec-3441 is blocking exec-1863 because it's holding a lock on locked <0x00007f3a4650d6a8> (a net.sf.hibernate.cache.ReadWriteCache):

"http-nio-8080-exec-3441" #233418 daemon prio=5 os_prio=0 tid=0x00007f38641be000 nid=0x1410d runnable [0x00007f35bbbf1000]
   java.lang.Thread.State: RUNNABLE
	at net.sf.hibernate.cache.ReadWriteCache.get(ReadWriteCache.java:69)
	- locked <0x00007f3a4650d6a8> (a net.sf.hibernate.cache.ReadWriteCache)
	at com.atlassian.confluence.cache.hibernate.SwitchableCacheConcurrencyStrategy.get(SwitchableCacheConcurrencyStrategy.java:29)
	at com.atlassian.confluence.cache.hibernate.ConfluenceCacheStrategy.get(ConfluenceCacheStrategy.java:49)
	at com.atlassian.hibernate.adapter.adapters.cache.EntityRegionAccessStrategyV5Adapter.get(EntityRegionAccessStrategyV5Adapter.java:51)
	at org.hibernate.engine.internal.CacheHelper.fromSharedCache(CacheHelper.java:32)
	at org.hibernate.engine.spi.BatchFetchQueue.isCached(BatchFetchQueue.java:226)
	at org.hibernate.engine.spi.BatchFetchQueue.getEntityBatch(BatchFetchQueue.java:201)
	at org.hibernate.loader.entity.plan.LegacyBatchingEntityLoaderBuilder$LegacyBatchingEntityLoader.load(LegacyBatchingEntityLoaderBuilder.java:91)
	at org.hibernate.persister.entity.AbstractEntityPersister.load(AbstractEntityPersister.java:4073)
	at org.hibernate.event.internal.DefaultLoadEventListener.loadFromDatasource(DefaultLoadEventListener.java:508)
	at org.hibernate.event.internal.DefaultLoadEventListener.doLoad(DefaultLoadEventListener.java:478)
	at org.hibernate.event.internal.DefaultLoadEventListener.load(DefaultLoadEventListener.java:219)
	at org.hibernate.event.internal.DefaultLoadEventListener.doOnLoad(DefaultLoadEventListener.java:116)
	at org.hibernate.event.internal.DefaultLoadEventListener.onLoad(DefaultLoadEventListener.java:89)
	at org.hibernate.internal.SessionImpl.fireLoad(SessionImpl.java:1230)
	at org.hibernate.internal.SessionImpl.immediateLoad(SessionImpl.java:1088)
	at org.hibernate.proxy.AbstractLazyInitializer.initialize(AbstractLazyInitializer.java:155)
	at org.hibernate.proxy.AbstractLazyInitializer.getImplementation(AbstractLazyInitializer.java:259)
	at org.hibernate.proxy.pojo.javassist.JavassistLazyInitializer.invoke(JavassistLazyInitializer.java:73)
	at com.atlassian.hibernate.adapter.proxy.JavassistLazyInitializer_ImplementV2Proxy.invoke(JavassistLazyInitializer_ImplementV2Proxy.java:68)
	at com.atlassian.confluence.user.ConfluenceUserImpl_$$_jvstd8c_3.getName(ConfluenceUserImpl_$$_jvstd8c_3.java)
	at com.atlassian.confluence.security.ContentPermission.compareTo(ContentPermission.java:253)
	at com.atlassian.confluence.security.ContentPermission.compareTo(ContentPermission.java:31)
	at java.util.TreeMap.put(TreeMap.java:568)
	at java.util.TreeSet.add(TreeSet.java:255)
	at java.util.AbstractCollection.addAll(AbstractCollection.java:344)
	at java.util.TreeSet.addAll(TreeSet.java:312)
	at org.hibernate.collection.internal.PersistentSet.endRead(PersistentSet.java:327)
	at org.hibernate.engine.loading.internal.CollectionLoadContext.endLoadingCollection(CollectionLoadContext.java:234)
	at org.hibernate.engine.loading.internal.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:221)
	at org.hibernate.engine.loading.internal.CollectionLoadContext.endLoadingCollections(CollectionLoadContext.java:194)
	at org.hibernate.loader.plan.exec.process.internal.CollectionReferenceInitializerImpl.endLoading(CollectionReferenceInitializerImpl.java:154)
	at org.hibernate.loader.plan.exec.process.internal.AbstractRowReader.finishLoadingCollections(AbstractRowReader.java:249)
	at org.hibernate.loader.plan.exec.process.internal.AbstractRowReader.finishUp(AbstractRowReader.java:212)
	at org.hibernate.loader.plan.exec.process.internal.ResultSetProcessorImpl.extractResults(ResultSetProcessorImpl.java:133)
	at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:122)
	at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:86)
	at org.hibernate.loader.collection.plan.AbstractLoadPlanBasedCollectionInitializer.initialize(AbstractLoadPlanBasedCollectionInitializer.java:87)
	at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:688)
	at org.hibernate.event.internal.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:75)
	at org.hibernate.internal.SessionImpl.initializeCollection(SessionImpl.java:2176)
	at org.hibernate.collection.internal.AbstractPersistentCollection$4.doWork(AbstractPersistentCollection.java:565)
	at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:247)
	at org.hibernate.collection.internal.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:561)
	at org.hibernate.collection.internal.AbstractPersistentCollection.forceInitialization(AbstractPersistentCollection.java:729)
	at org.hibernate.engine.internal.StatefulPersistenceContext.initializeNonLazyCollections(StatefulPersistenceContext.java:926)
	at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:149)
	at org.hibernate.loader.plan.exec.internal.AbstractLoadPlanBasedLoader.executeLoad(AbstractLoadPlanBasedLoader.java:86)
	at org.hibernate.loader.collection.plan.AbstractLoadPlanBasedCollectionInitializer.initialize(AbstractLoadPlanBasedCollectionInitializer.java:87)
	at org.hibernate.persister.collection.AbstractCollectionPersister.initialize(AbstractCollectionPersister.java:688)
	at org.hibernate.event.internal.DefaultInitializeCollectionEventListener.onInitializeCollection(DefaultInitializeCollectionEventListener.java:75)
	at org.hibernate.internal.SessionImpl.initializeCollection(SessionImpl.java:2176)
	at org.hibernate.collection.internal.AbstractPersistentCollection$4.doWork(AbstractPersistentCollection.java:565)
	at org.hibernate.collection.internal.AbstractPersistentCollection.withTemporarySessionIfNeeded(AbstractPersistentCollection.java:247)
	at org.hibernate.collection.internal.AbstractPersistentCollection.initialize(AbstractPersistentCollection.java:561)
	at org.hibernate.collection.internal.AbstractPersistentCollection.read(AbstractPersistentCollection.java:132)
	at org.hibernate.collection.internal.PersistentBag.iterator(PersistentBag.java:277)
	at com.atlassian.confluence.core.ContentEntityObject.getContentPermissionSet(ContentEntityObject.java:1083)
	at com.atlassian.confluence.core.DefaultContentPermissionManager.hasContentLevelPermissionIgnoreInherited(DefaultContentPermissionManager.java:214)
	at com.atlassian.confluence.core.DefaultContentPermissionManager.hasPermittedChildrenIgnoreInheritedPermissions(DefaultContentPermissionManager.java:445)
	...
	at com.sun.proxy.$Proxy102.hasPermittedChildrenIgnoreInheritedPermissions(Unknown Source)
	at com.atlassian.confluence.pages.actions.ChildrenAction.hasPermittedChildren(ChildrenAction.java:59)
	at com.adaptavist.confluence.naturalchildren.NaturalChildrenAction.hasPermittedChildren(NaturalChildrenAction.java:238)
	...
	at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:270)
	at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:262)
	at org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:507)
	at org.apache.velocity.runtime.parser.node.ASTExpression.value(ASTExpression.java:71)
	at org.apache.velocity.runtime.parser.node.ASTSetDirective.render(ASTSetDirective.java:142)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:212)
	at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:247)
	at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:175)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.directive.Foreach.performIteration(Foreach.java:393)
	at org.apache.velocity.runtime.directive.Foreach.render(Foreach.java:316)
	at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:175)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:212)
	at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:247)
	at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:175)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.directive.Foreach.performIteration(Foreach.java:393)
	at org.apache.velocity.runtime.directive.Foreach.render(Foreach.java:316)
	at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:175)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:212)
	at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:247)
	at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:175)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.directive.Foreach.performIteration(Foreach.java:393)
	at org.apache.velocity.runtime.directive.Foreach.render(Foreach.java:316)
	at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:175)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:87)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:212)
	at org.apache.velocity.runtime.directive.RuntimeMacro.render(RuntimeMacro.java:247)
	at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:175)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.directive.Foreach.performIteration(Foreach.java:393)
	at org.apache.velocity.runtime.directive.Foreach.render(Foreach.java:316)
	at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:175)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:72)
	at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:336)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:106)
	at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:336)
	at org.apache.velocity.Template.merge(Template.java:328)
	at org.apache.velocity.Template.merge(Template.java:235)
	at com.opensymphony.webwork.dispatcher.VelocityResult.doExecute(VelocityResult.java:91)
	at com.atlassian.xwork.results.ProfiledVelocityResult.doExecute(ProfiledVelocityResult.java:21)
	at com.atlassian.confluence.setup.webwork.EncodingVelocityResult.doExecute(EncodingVelocityResult.java:43)
	at com.opensymphony.webwork.dispatcher.WebWorkResultSupport.execute(WebWorkResultSupport.java:116)
	at com.opensymphony.xwork.DefaultActionInvocation.executeResult(DefaultActionInvocation.java:263)
	...
	at com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter.lambda$doFilter$3(VCacheRequestContextFilter.java:44)
	at com.atlassian.confluence.impl.vcache.VCacheRequestContextFilter$$Lambda$681/1734476358.perform(Unknown Source)
	at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContextInternal(VCacheRequestContextManager.java:87)
	at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:71)
	...
 	at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1132)
 	at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:684)
 	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1539)
 	at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1495)
 	- locked <0x00007f3ad35fbaf8> (a org.apache.tomcat.util.net.NioChannel)
 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
 	at java.lang.Thread.run(Thread.java:748)

    Locked ownable synchronizers:
 	- <0x00007f3b0161ecb8> (a java.util.concurrent.ThreadPoolExecutor$Worker)

Looking at the http-nio-8080-exec-1863, this thread obtained net.sf.hibernate.cache.ReadWriteCache's lock & did not release it. Due to that 1596 threads are BLOCKED:

"http-nio-8080-exec-1863" #57426 daemon prio=5 os_prio=0 tid=0x00007f3858606000 nid=0x45eb waiting for monitor entry [0x00007f31ce9c3000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at net.sf.hibernate.cache.ReadWriteCache.get(ReadWriteCache.java:69)
	- waiting to lock <0x00007f3a4650d6a8> (a net.sf.hibernate.cache.ReadWriteCache)
	at com.atlassian.confluence.cache.hibernate.SwitchableCacheConcurrencyStrategy.get(SwitchableCacheConcurrencyStrategy.java:29)
	at com.atlassian.confluence.cache.hibernate.ConfluenceCacheStrategy.get(ConfluenceCacheStrategy.java:49)
	...
	at com.atlassian.confluence.core.ContentEntityObject.getContentPermissionSet(ContentEntityObject.java:1083)
	at com.atlassian.confluence.core.DefaultContentPermissionManager.hasContentLevelPermissionIgnoreInherited(DefaultContentPermissionManager.java:214)
	at com.atlassian.confluence.core.DefaultContentPermissionManager.hasContentLevelPermission(DefaultContentPermissionManager.java:202)
	...
	at com.atlassian.confluence.security.delegate.PagePermissionsDelegate.hasContentLevelViewPermission(PagePermissionsDelegate.java:124)
	at com.atlassian.confluence.security.delegate.PagePermissionsDelegate.canView(PagePermissionsDelegate.java:21)
	at com.atlassian.confluence.security.delegate.TargetToLatestVersionDecorator.canView(TargetToLatestVersionDecorator.java:24)
	at com.atlassian.confluence.security.delegate.SharedAccessInterceptor$$Lambda$1007/1154506160.test(Unknown Source)
	at com.atlassian.confluence.security.delegate.SharedAccessInterceptor.checkAccess(SharedAccessInterceptor.java:88)
	at com.atlassian.confluence.security.delegate.SharedAccessInterceptor.canView(SharedAccessInterceptor.java:26)
	at com.atlassian.confluence.security.Permission$1.checkAgainst(Permission.java:17)
	at com.atlassian.confluence.security.DefaultPermissionManager.hasPermissionNoExemptions(DefaultPermissionManager.java:85)
	at com.atlassian.confluence.security.DefaultPermissionManager.hasPermission(DefaultPermissionManager.java:42)
	at com.atlassian.confluence.security.DefaultPermissionManager.getPermittedEntities(DefaultPermissionManager.java:126)
	...
	at com.adaptavist.confluence.naturalchildren.NaturalChildrenAction.getAllPermittedStartPages(NaturalChildrenAction.java:262)

Heap Dump

In this case, the heap was filled by org.hibernate.engine.internal.StatefulPersistenceContext objects (at around 12.6GB and 14.6GB out of 28GB of heap. Please refer to the screenshot below:

A big chunk of the org.hibernate.engine.internal.StatefulPersistenceContext objects are associated to TaskThread, in which are trying to retrieve content permission as shown in the sample thread details' snippet below:

http-nio-8080-exec-3592
  at net.sf.hibernate.cache.ReadWriteCache.get(Ljava/lang/Object;J)Ljava/lang/Object; (ReadWriteCache.java:69)
  at com.atlassian.confluence.cache.hibernate.SwitchableCacheConcurrencyStrategy.get(Ljava/lang/Object;J)Ljava/lang/Object; (SwitchableCacheConcurrencyStrategy.java:29)
  at com.atlassian.confluence.cache.hibernate.ConfluenceCacheStrategy.get(Ljava/lang/Object;J)Ljava/lang/Object; (ConfluenceCacheStrategy.java:49)
  ...
  at com.atlassian.confluence.user.ConfluenceUserImpl_$$_jvstd8c_3.getName()Ljava/lang/String; (Unknown Source)
  at com.atlassian.confluence.security.ContentPermission.compareTo(Lcom/atlassian/confluence/security/ContentPermission;)I (ContentPermission.java:253)
  at com.atlassian.confluence.security.ContentPermission.compareTo(Ljava/lang/Object;)I (ContentPermission.java:31)
  at java.util.TreeMap.put(Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object; (TreeMap.java:568)
...

Logs

Logs may contain entries like the following:

2018-04-18 23:12:35,776 WARN [http-nio-8080-exec-883] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1524093066939, threadId=1123, threadName='http-nio-8080-exec-883', userId='abc123', type='web-request', summary='/plugins/pagetree/naturalchildren.action?decorator=none&excerpt=false&sort=position&reverse=false&disableLinks=false&expandCurrent=true&hasRoot=true&pageId=348115741&treeId=0&startDepth=0&mobile=false&ancestors=366428612&ancestors=348115741&treePageId=366428767'}
 -- url: /confluence/plugins/pagetree/naturalchildren.action

Cache Statistics

Looking at the relted cache statistics we see a very high (100%) capacity utilization but a very low effectiveness (5% - 17%) for the following caches:

Content Permissions
Space Permissions (by ID)
Content Properties
Embedded Crowd Internal User
Content Objects
Page Children

As seen in this example:

Synopsis

In this case, there was a permissions lookup triggered by a plugin. This lookup tried to retrieve the data from cache, but a concurrent cache lock issue occurred during the cache eviction. In the above example when the lock was obtained, all other threads needing a lock on net.sf.hibernate.cache.ReadWriteCache had to wait for the first thread to finish. Due to this a large amount of threads built up waiting on the same task. This buildup of blocked threads resulted in very few threads being available to perform other actions and resulted in a performance impact to Confluence as users were not able to use the site.

This was discovered to be caused by a large space changing the sidebar display from Child Pages to Page Tree thus resulting in a larger number of pages listed in the sidebar, leading to more permissions lookups.

Workarounds

The workarounds for this issue until the code is optimized, is to either tune the Confluence cache, and/or discontinue the use of the Page Tree view in large space's sidebars.

This query will identify which space keys use each sidebar configuration:

# select bandanacontext, bandanavalue from bandana where bandanakey = 'sidebar.nav-type';
 bandanacontext |        bandanavalue
----------------+----------------------------
 ds             | <string>page-tree</string>
 TEST           | <string>pages</string>
(2 rows)

page-tree = Page Tree view
pages = Child Pages view

PLEASE READ: As with all recommendations made by Atlassian Support, please follow best practices for Change Management and test and validate these settings in a Test/Development and Staging environment prior to rolling any changes into a Production environment. This is to validate these changes and ensure that they will function will within your infrastructure prior to placing these changes in production.

PLEASE READ: There is additional advice given at the end of this comment which should be read before you move forward with tuning the caches in the environment. It is better to start with the infrastructure components to determine if any adjustments are necessary before adjusting the caches within Confluence.

You should only consider increasing the caches below, when the capacity utilization is high (at or near 100%) and the effectiveness percentage is low (under 20%). That is to say, if capacity utilization is at 100% and the effectiveness percentage is very low ~(1 - 20%), those caches are candidates for alteration of the cache settings to increase the number of objects stored in memory.

Based upon the Cache view shown above, the candidate objects for cache alteration are:

Content Permissions
Space Permissions (by ID)
Content Properties
Embedded Crowd Internal User
Content Objects
Page Children

Any other objects should not be modified. If you are not sure whether or how to proceed, please contact support before making any changes to the cache.

Suggestions on how large the size should be adjusted

The basic formula to apply for a conservative approach would be to mark this as follows:

New Cache Size = Current Cache Size x 1.5

This basically means that the cache objects would be 150% of their current size. This could then be tested to determine how effective the cache would be in response to these changes. You may need to iterate through several adjustment and observation periods to make these caches more effective.

Adjusting the Cache

To change the size of a cache

Go to > General Configuration > Cache Management.
Choose Show Advanced View.
Choose Adjust Size next to the cache you want to change.

To modify other parameters you can modify the cache configuration files manually.

Cache configurations are stored in <confluence-home>/shared-home/config/cache-settings-overrides.properties
For Confluence Data Center (clustered) it can be found in <confluence-shared-home>/config/cache-settings-overrides.properties (in the shared home directory for the cluster).

Additional Advice

Depending on the database you are using within the environment, it may also be worthwhile to run any statistics, tuning or performance analysis provided by the native database tools for that repository. Doing so may provide you with additional instructions that may provide significant performance increases without resorting to altering the caches in the instance. I recommend involving your DBA as a rebuild of the indexes may provide significant performance improvement in itself.

Reference

Recommended reading: Cache Performance Tuning

Attachments

- Sort By Name
- Sort By Date
- Ascending
- Descending
- Thumbnails
- List

cache.png
283 kB
19/Apr/2018 8:20 PM
cache2.png
430 kB
19/Apr/2018 8:20 PM
heap1.png
124 kB
19/Apr/2018 8:29 PM
page-tree-show-all.png
345 kB
11/Feb/2020 3:28 AM

Issue Links

is related to

CONFSERVER-56391 Loading the "Reorder Pages" space tool page for very large spaces as a non site-admin user causes CPU spike

Closed

CONFSERVER-56068 childPagesContextualNav impacts Confluence performance

Short Term Backlog

relates to

CONFSERVER-43752 Slow page tree performance in the sidebar for large spaces in Confluence

Closed

CONFSERVER-20820 Improve Pagetree and Children macro performance on large spaces, PermissionManager.getPermittedEntities()

Closed

causes: PSR-251 Loading...

is blocked by: PSR-84 Loading...

mentioned in: Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...; Page Loading...

(1 is blocked by, 47 mentioned in)

Activity

People

Assignee:: George Lipatov

Reporter:: Justin W.

Votes:: 29 Vote for this issue

Watchers:: 54 Start watching this issue

Dates

Created:: 19/Apr/2018 7:43 PM

Updated:: 28/Apr/2023 1:29 AM

Resolved:: 06/Feb/2020 12:45 AM