Permission reindexing consumes excessive CPU on comment-heavy page trees due to repeated Hibernate autoflush checks

XMLWordPrintable

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Medium
    • None
    • Affects Version/s: 10.2.18, 10.2.19
    • None
    • 4
    • Severity 3 - Minor
    • 54

      Issue Summary

      For a large data set, ContentTreePermissionReindexEventBackgroundSender()->process() consumes 100% CPU busy and can take a long time (days or weeks) to run due to calls through findSubscriptionsByContextKeyAndState() performing an autoFlushIfRequired.

      Steps to Reproduce

      1. Install a clustered Confluence instance with multiple nodes
      2. Setup a data set with the following structure:
      3. Create a new "Base Page"
      4. Directly under Base Page
        • Page1 (800 pages directly under this)
          • Under one of the new child pages, create a deep page tree of 1200 pages deep
        • Page2 (800 pages directly under this)
          • Under one of the new child pages, create a deep page tree of 1200 pages deep
        • Page3 (800 pages directly under this)
          • Under one of the new child pages, create a deep page tree of 1200 pages deep
      5. Every single page above should have 1000's of comments added to the page
      6. The aim is to create a large enough data set so that ContentTreePermissionReindexEventBackgroundSender->process() processing takes a long time.
      7. Change the permission on Base Page
        • This will result in a ContentTreePermissionReindexEventBackgroundSender added to table background_job

      Expected Results

      The ContentTreePermissionReindexEventBackgroundSender job shouldn't take days/weeks to complete due to calls through findSubscriptionsByContextKeyAndState() performing a NOOP autoFlushIfRequired.

      Actual Results

      1. Querying background_job shows a very old creation_time indicating the job has been running for a long time. e.g.
        select * FROM background_job where type = 'ContentTreePermissionReindexEventBackgroundSender' order by creation_time;
        
        17661954	ContentTreePermissionReindexEventBackgroundSender	(null)	Sending ContentTreePermissionReindexEvent for all descendants of content entity object with id 163926 and title test	2026-10-12 02:34:28	3	0	2026-10-12 01:53:23	2026-10-12 02:34:28	1821758	{"pageIdsToProcess":[4959814,4963555,4967169,4970810,4974409,4977946,5079988,5083536,5087136,5090718,5094355,5097865,5101394,5105022,5108714,5210701,5214408]}
        
      2. Taking CPU & Thread dumps will show caesium thread consuming continuous 100% CPU busy for the ContentTreePermissionReindexEventBackgroundSender job predominantly performing an autoFlushIfRequired in hibernate layer invoked from findSubscriptionsByContextKeyAndState
        "Caesium-1-2" #538 [1131328] daemon prio=1 os_prio=0 cpu=106723856.81ms elapsed=392999.45s tid=0x00007ee53d9940e0 nid=1131328 runnable  [0x00007ee57faf6000]
           java.lang.Thread.State: RUNNABLE
        	at org.hibernate.engine.internal.StatefulPersistenceContext.removeChildParent(StatefulPersistenceContext.java:1800)
        	at org.hibernate.engine.internal.Cascade.cascadeToOne(Cascade.java:513)
        	at org.hibernate.engine.internal.Cascade.cascadeAssociation(Cascade.java:434)
        	at org.hibernate.engine.internal.Cascade.cascadeProperty(Cascade.java:220)
        	at org.hibernate.engine.internal.Cascade.cascade(Cascade.java:153)
        	at org.hibernate.event.internal.AbstractFlushingEventListener.cascadeOnFlush(AbstractFlushingEventListener.java:159)
        	at org.hibernate.event.internal.AbstractFlushingEventListener.prepareEntityFlushes(AbstractFlushingEventListener.java:149)
        	at org.hibernate.event.internal.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:82)
        	at org.hibernate.event.internal.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:50)
        	at org.hibernate.internal.SessionImpl$$Lambda/0x00007f3189bd2048.accept(Unknown Source)
        	at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:107)
        	at org.hibernate.internal.SessionImpl.autoFlushIfRequired(SessionImpl.java:1372)
        	at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1452)
        	at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1649)
        	at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1617)
        	at org.hibernate.query.Query.getResultList(Query.java:165)
        	at org.hibernate.query.criteria.internal.compile.CriteriaQueryTypeQueryAdapter.getResultList(CriteriaQueryTypeQueryAdapter.java:76)
        	at org.springframework.data.jpa.repository.query.JpaQueryExecution$CollectionExecution.doExecute(JpaQueryExecution.java:132)
        	at org.springframework.data.jpa.repository.query.JpaQueryExecution.execute(JpaQueryExecution.java:95)
        	at org.springframework.data.jpa.repository.query.AbstractJpaQuery.doExecute(AbstractJpaQuery.java:160)
        	at org.springframework.data.jpa.repository.query.AbstractJpaQuery.execute(AbstractJpaQuery.java:148)
        ...
        	at org.springframework.data.repository.core.support.RepositoryMethodInvoker.doInvoke(RepositoryMethodInvoker.java:170)
        ...
        	at org.springframework.data.repository.core.support.QueryExecutorMethodInterceptor.doInvoke(QueryExecutorMethodInterceptor.java:170)
        ...
        	at jdk.proxy4.$Proxy326.findSubscriptionsByContextKeyAndState(jdk.proxy4/Unknown Source)
        	at com.atlassian.confluence.impl.incrementalsync.IncrementalSyncSubscriptionRepository.findSubscriptionsByContextKeyAndState(IncrementalSyncSubscriptionRepository.java:50)
        ...
        	at jdk.proxy4.$Proxy326.findSubscriptionsByContextKeyAndState(jdk.proxy4/Unknown Source)
        	at com.atlassian.confluence.impl.incrementalsync.DefaultIncrementalSyncSubscriptionManager.getSubscriptionsByContext(DefaultIncrementalSyncSubscriptionManager.java:75)
        	at com.atlassian.confluence.impl.incrementalsync.CachingIncrementalSyncSubscriptionManager.getSubscriptionsByContext(CachingIncrementalSyncSubscriptionManager.java:69)
        ...
        	at jdk.proxy4.$Proxy327.getSubscriptionsByContext(jdk.proxy4/Unknown Source)
        	at com.atlassian.confluence.internal.api.impl.service.incrementalsync.journal.ServerSubscriptionJournalStateStore.getAvailableSubscriptions(ServerSubscriptionJournalStateStore.java:96)
        ...
        	at com.atlassian.confluence.internal.api.impl.service.incrementalsync.contentmanagement.ContentManagementRecorder.executeCommand(ContentManagementRecorder.java:244)
        	at com.atlassian.confluence.internal.api.impl.service.incrementalsync.AbstractRecorder.lambda$handleEvent$0(AbstractRecorder.java:33)
        	at com.atlassian.confluence.internal.api.impl.service.incrementalsync.AbstractRecorder$$Lambda/0x00007f318ca922d8.run(Unknown Source)
        	at com.atlassian.confluence.internal.api.impl.service.incrementalsync.AbstractRecorder.lambda$executeWithTransaction$3(AbstractRecorder.java:50)
        	at com.atlassian.confluence.internal.api.impl.service.incrementalsync.AbstractRecorder$$Lambda/0x00007f318ca92500.doInTransaction(Unknown Source)
        	at com.atlassian.sal.core.transaction.HostContextTransactionTemplate$1.doInTransaction(HostContextTransactionTemplate.java:21)
        	at com.atlassian.sal.spring.component.SpringHostContextAccessor.lambda$doInTransaction$0(SpringHostContextAccessor.java:72)
        	at com.atlassian.sal.spring.component.SpringHostContextAccessor$$Lambda/0x00007f318af9fd38.doInTransaction(Unknown Source)
        ...
        	at com.atlassian.sal.spring.component.SpringHostContextAccessor.doInTransaction(SpringHostContextAccessor.java:70)
        ...
        	at com.atlassian.confluence.internal.api.impl.service.incrementalsync.AbstractRecorder.executeWithTransaction(AbstractRecorder.java:49)
        	at com.atlassian.confluence.internal.api.impl.service.incrementalsync.AbstractRecorder.handleEvent(AbstractRecorder.java:33)
        	at com.atlassian.confluence.internal.api.impl.service.incrementalsync.contentmanagement.ContentManagementRecorder.onContentTreePermissionReindexEvent(ContentManagementRecorder.java:74)
        ..
        	at com.atlassian.confluence.impl.content.ContentTreePermissionReindexEventBackgroundSender.sendContentTreePermissionReindexEvent(ContentTreePermissionReindexEventBackgroundSender.java:174)
        	at com.atlassian.confluence.impl.content.ContentTreePermissionReindexEventBackgroundSender.reindexContentAndComments(ContentTreePermissionReindexEventBackgroundSender.java:169)
        	at com.atlassian.confluence.impl.content.ContentTreePermissionReindexEventBackgroundSender$$Lambda/0x00007f545978b418.accept(Unknown Source)
        	at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:423)
        	at com.atlassian.confluence.impl.content.ContentTreePermissionReindexEventBackgroundSender.processNextBatchOfPages(ContentTreePermissionReindexEventBackgroundSender.java:136)
        	at com.atlassian.confluence.impl.content.ContentTreePermissionReindexEventBackgroundSender.lambda$process$0(ContentTreePermissionReindexEventBackgroundSender.java:98)
        	at com.atlassian.confluence.impl.content.ContentTreePermissionReindexEventBackgroundSender$$Lambda/0x00007f545978a8b0.doInTransaction(Unknown Source)
        	at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:140)
        	at com.atlassian.confluence.impl.content.ContentTreePermissionReindexEventBackgroundSender.doInTransaction(ContentTreePermissionReindexEventBackgroundSender.java:180)
        	at com.atlassian.confluence.impl.content.ContentTreePermissionReindexEventBackgroundSender.process(ContentTreePermissionReindexEventBackgroundSender.java:98)
        	at com.atlassian.confluence.impl.backgroundjob.BackgroundJobService.runBackgroundProcess(BackgroundJobService.java:268)
        	at com.atlassian.confluence.impl.backgroundjob.BackgroundJobService.processNextJobs(BackgroundJobService.java:175)
        ..
        

      Workaround

      Currently there is no known workaround for this behavior.

              Assignee:
              Unassigned
              Reporter:
              Eric L
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: