-
Type:
Bug
-
Resolution: Unresolved
-
Priority:
Medium
-
None
-
Affects Version/s: 10.2.18, 10.2.19
-
Component/s: Search - Indexing, Server - Jobs
-
None
-
4
-
Severity 3 - Minor
-
54
Issue Summary
For a large data set, ContentTreePermissionReindexEventBackgroundSender()->process() consumes 100% CPU busy and can take a long time (days or weeks) to run due to calls through findSubscriptionsByContextKeyAndState() performing an autoFlushIfRequired.
Steps to Reproduce
- Install a clustered Confluence instance with multiple nodes
- Setup a data set with the following structure:
- Create a new "Base Page"
- Directly under Base Page
- Page1 (800 pages directly under this)
- Under one of the new child pages, create a deep page tree of 1200 pages deep
- Page2 (800 pages directly under this)
- Under one of the new child pages, create a deep page tree of 1200 pages deep
- Page3 (800 pages directly under this)
- Under one of the new child pages, create a deep page tree of 1200 pages deep
- Page1 (800 pages directly under this)
Every single page above should have 1000's of comments added to the page
The aim is to create a large enough data set so that ContentTreePermissionReindexEventBackgroundSender->process() processing takes a long time.- Change the permission on Base Page
- This will result in a ContentTreePermissionReindexEventBackgroundSender added to table background_job
Expected Results
The ContentTreePermissionReindexEventBackgroundSender job shouldn't take days/weeks to complete due to calls through findSubscriptionsByContextKeyAndState() performing a NOOP autoFlushIfRequired.
Actual Results
- Querying background_job shows a very old creation_time indicating the job has been running for a long time. e.g.
select * FROM background_job where type = 'ContentTreePermissionReindexEventBackgroundSender' order by creation_time;
17661954 ContentTreePermissionReindexEventBackgroundSender (null) Sending ContentTreePermissionReindexEvent for all descendants of content entity object with id 163926 and title test 2026-10-12 02:34:28 3 0 2026-10-12 01:53:23 2026-10-12 02:34:28 1821758 {"pageIdsToProcess":[4959814,4963555,4967169,4970810,4974409,4977946,5079988,5083536,5087136,5090718,5094355,5097865,5101394,5105022,5108714,5210701,5214408]} - Taking CPU & Thread dumps will show caesium thread consuming continuous 100% CPU busy for the ContentTreePermissionReindexEventBackgroundSender job predominantly performing an autoFlushIfRequired in hibernate layer invoked from findSubscriptionsByContextKeyAndState
"Caesium-1-2" #538 [1131328] daemon prio=1 os_prio=0 cpu=106723856.81ms elapsed=392999.45s tid=0x00007ee53d9940e0 nid=1131328 runnable [0x00007ee57faf6000] java.lang.Thread.State: RUNNABLE at org.hibernate.engine.internal.StatefulPersistenceContext.removeChildParent(StatefulPersistenceContext.java:1800) at org.hibernate.engine.internal.Cascade.cascadeToOne(Cascade.java:513) at org.hibernate.engine.internal.Cascade.cascadeAssociation(Cascade.java:434) at org.hibernate.engine.internal.Cascade.cascadeProperty(Cascade.java:220) at org.hibernate.engine.internal.Cascade.cascade(Cascade.java:153) at org.hibernate.event.internal.AbstractFlushingEventListener.cascadeOnFlush(AbstractFlushingEventListener.java:159) at org.hibernate.event.internal.AbstractFlushingEventListener.prepareEntityFlushes(AbstractFlushingEventListener.java:149) at org.hibernate.event.internal.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:82) at org.hibernate.event.internal.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:50) at org.hibernate.internal.SessionImpl$$Lambda/0x00007f3189bd2048.accept(Unknown Source) at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:107) at org.hibernate.internal.SessionImpl.autoFlushIfRequired(SessionImpl.java:1372) at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1452) at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1649) at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1617) at org.hibernate.query.Query.getResultList(Query.java:165) at org.hibernate.query.criteria.internal.compile.CriteriaQueryTypeQueryAdapter.getResultList(CriteriaQueryTypeQueryAdapter.java:76) at org.springframework.data.jpa.repository.query.JpaQueryExecution$CollectionExecution.doExecute(JpaQueryExecution.java:132) at org.springframework.data.jpa.repository.query.JpaQueryExecution.execute(JpaQueryExecution.java:95) at org.springframework.data.jpa.repository.query.AbstractJpaQuery.doExecute(AbstractJpaQuery.java:160) at org.springframework.data.jpa.repository.query.AbstractJpaQuery.execute(AbstractJpaQuery.java:148) ... at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:170) ... at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:170) ... at jdk.proxy4.$Proxy326.findSubscriptionsByContextKeyAndState(jdk.proxy4/Unknown Source) at com.atlassian.confluence.impl.incrementalsync.IncrementalSyncSubscriptionRepository.findSubscriptionsByContextKeyAndState(IncrementalSyncSubscriptionRepository.java:50) ... at jdk.proxy4.$Proxy326.findSubscriptionsByContextKeyAndState(jdk.proxy4/Unknown Source) at com.atlassian.confluence.impl.incrementalsync.DefaultIncrementalSyncSubscriptionManager.getSubscriptionsByContext(DefaultIncrementalSyncSubscriptionManager.java:75) at com.atlassian.confluence.impl.incrementalsync.CachingIncrementalSyncSubscriptionManager.getSubscriptionsByContext(CachingIncrementalSyncSubscriptionManager.java:69) ... at jdk.proxy4.$Proxy327.getSubscriptionsByContext(jdk.proxy4/Unknown Source) at com.atlassian.confluence.internal.api.impl.service.incrementalsync.journal.ServerSubscriptionJournalStateStore.getAvailableSubscriptions(ServerSubscriptionJournalStateStore.java:96) ... at com.atlassian.confluence.internal.api.impl.service.incrementalsync.contentmanagement.ContentManagementRecorder.executeCommand(ContentManagementRecorder.java:244) at com.atlassian.confluence.internal.api.impl.service.incrementalsync.AbstractRecorder.lambda$handleEvent$0(AbstractRecorder.java:33) at com.atlassian.confluence.internal.api.impl.service.incrementalsync.AbstractRecorder$$Lambda/0x00007f318ca922d8.run(Unknown Source) at com.atlassian.confluence.internal.api.impl.service.incrementalsync.AbstractRecorder.lambda$executeWithTransaction$3(AbstractRecorder.java:50) at com.atlassian.confluence.internal.api.impl.service.incrementalsync.AbstractRecorder$$Lambda/0x00007f318ca92500.doInTransaction(Unknown Source) at com.atlassian.sal.core.transaction.HostContextTransactionTemplate$1.doInTransaction(HostContextTransactionTemplate.java:21) at com.atlassian.sal.spring.component.SpringHostContextAccessor.lambda$doInTransaction$0(SpringHostContextAccessor.java:72) at com.atlassian.sal.spring.component.SpringHostContextAccessor$$Lambda/0x00007f318af9fd38.doInTransaction(Unknown Source) ... at com.atlassian.sal.spring.component.SpringHostContextAccessor.doInTransaction(SpringHostContextAccessor.java:70) ... at com.atlassian.confluence.internal.api.impl.service.incrementalsync.AbstractRecorder.executeWithTransaction(AbstractRecorder.java:49) at com.atlassian.confluence.internal.api.impl.service.incrementalsync.AbstractRecorder.handleEvent(AbstractRecorder.java:33) at com.atlassian.confluence.internal.api.impl.service.incrementalsync.contentmanagement.ContentManagementRecorder.onContentTreePermissionReindexEvent(ContentManagementRecorder.java:74) .. at com.atlassian.confluence.impl.content.ContentTreePermissionReindexEventBackgroundSender.sendContentTreePermissionReindexEvent(ContentTreePermissionReindexEventBackgroundSender.java:174) at com.atlassian.confluence.impl.content.ContentTreePermissionReindexEventBackgroundSender.reindexContentAndComments(ContentTreePermissionReindexEventBackgroundSender.java:169) at com.atlassian.confluence.impl.content.ContentTreePermissionReindexEventBackgroundSender$$Lambda/0x00007f545978b418.accept(Unknown Source) at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:423) at com.atlassian.confluence.impl.content.ContentTreePermissionReindexEventBackgroundSender.processNextBatchOfPages(ContentTreePermissionReindexEventBackgroundSender.java:136) at com.atlassian.confluence.impl.content.ContentTreePermissionReindexEventBackgroundSender.lambda$process$0(ContentTreePermissionReindexEventBackgroundSender.java:98) at com.atlassian.confluence.impl.content.ContentTreePermissionReindexEventBackgroundSender$$Lambda/0x00007f545978a8b0.doInTransaction(Unknown Source) at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140) at com.atlassian.confluence.impl.content.ContentTreePermissionReindexEventBackgroundSender.doInTransaction(ContentTreePermissionReindexEventBackgroundSender.java:180) at com.atlassian.confluence.impl.content.ContentTreePermissionReindexEventBackgroundSender.process(ContentTreePermissionReindexEventBackgroundSender.java:98) at com.atlassian.confluence.impl.backgroundjob.BackgroundJobService.runBackgroundProcess(BackgroundJobService.java:268) at com.atlassian.confluence.impl.backgroundjob.BackgroundJobService.processNextJobs(BackgroundJobService.java:175) ..
Workaround
Currently there is no known workaround for this behavior.
- is related to
-
CONFSERVER-104495 Cluster nodes can concurrently process the same permission-reindex batch, creating duplicate journal entries
-
- Gathering Impact
-