Sites and spaces with a large number of attachments can't be exported from Confluence

XMLWordPrintable

    • 1
    • Severity 2 - Major

      Summary

      On export (whether to XML, PDF, or HTML) Confluence goes through and fetches each attachment individually from the Media API. On an instance with a high-enough number of attachments, this means that we can never manage to export because the process takes so long that Confluence is restarted for upgrades before it completes.

      Steps to reproduce

      1. Add a massive number of attachments to a space (the support case I'm dealing with has ~160k in a single space, ~4.5k pages).
      2. Try to export by any means (PDF, XML, HTML).

      Notes

      Confluence hovers around 100-150% CPU, and the thread using up all the CPU is this:

      "Caesium-1-1" #431 daemon prio=5 os_prio=0 tid=0x0000000009e97000 nid=0xee0 runnable [0x00007f453f805000]
   java.lang.Thread.State: RUNNABLE
        at org.apache.log4j.Category.getEffectiveLevel(Category.java:442)
        at org.apache.log4j.Category.isEnabledFor(Category.java:751)
        at org.jboss.logging.Log4jLogger.doLog(Log4jLogger.java:41)
        at org.jboss.logging.Logger.trace(Logger.java:128)
        at org.jboss.logging.DelegatingBasicLogger.trace(DelegatingBasicLogger.java:59)
        at org.hibernate.event.internal.AbstractFlushingEventListener.flushCollections(AbstractFlushingEventListener.java:249)
        at org.hibernate.event.internal.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:86)
        at org.hibernate.event.internal.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:44)
        at org.hibernate.internal.SessionImpl.autoFlushIfRequired(SessionImpl.java:1366)
        at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1451)
        at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1426)
        at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1398)
        at com.atlassian.confluence.core.persistence.hibernate.HibernateObjectDao.lambda$findNamedQueryStringParams$5(HibernateObjectDao.java:381)
        at com.atlassian.confluence.core.persistence.hibernate.HibernateObjectDao$$Lambda$1059/1911612343.doInHibernate(Unknown Source)
        at org.springframework.orm.hibernate5.HibernateTemplate.doExecute(HibernateTemplate.java:359)
        at org.springframework.orm.hibernate5.HibernateTemplate.executeWithNativeSession(HibernateTemplate.java:326)
        at com.atlassian.confluence.core.persistence.hibernate.HibernateObjectDao.findNamedQueryStringParams(HibernateObjectDao.java:358)
        at com.atlassian.confluence.core.persistence.hibernate.HibernateObjectDao.findNamedQueryStringParams(HibernateObjectDao.java:331)
        at com.atlassian.confluence.pages.persistence.dao.hibernate.AbstractHibernateAttachmentDao.getLatestAttachment(AbstractHibernateAttachmentDao.java:209)
        at com.atlassian.confluence.pages.attachments.CachingAttachmentDao.lambda$getLatestAttachment$0(CachingAttachmentDao.java:81)
        at com.atlassian.confluence.pages.attachments.CachingAttachmentDao$$Lambda$1841/1591471292.get(Unknown Source)
        at com.atlassian.confluence.pages.attachments.AttachmentCache$$Lambda$1842/1234830727.get(Unknown Source)
        at com.atlassian.vcache.internal.core.metrics.TimedSupplier.get(TimedSupplier.java:32)
        at com.atlassian.vcache.internal.core.service.DefaultRequestCache.lambda$get$2(DefaultRequestCache.java:52)
        at com.atlassian.vcache.internal.core.service.DefaultRequestCache$$Lambda$989/1296042091.get(Unknown Source)
        at java.util.Optional.orElseGet(Optional.java:267)
        at com.atlassian.vcache.internal.core.service.DefaultRequestCache.get(DefaultRequestCache.java:50)
        at com.atlassian.vcache.internal.core.metrics.TimedLocalCacheOperations.get(TimedLocalCacheOperations.java:64)
        at com.atlassian.confluence.impl.vcache.metrics.vertigostartup.VertigoStartupMonitoringRequestCache.get(VertigoStartupMonitoringRequestCache.java:36)
        at com.atlassian.confluence.pages.attachments.AttachmentCache.get(AttachmentCache.java:52)
        at com.atlassian.confluence.pages.attachments.CachingAttachmentDao.getLatestAttachment(CachingAttachmentDao.java:77)
        at com.atlassian.confluence.pages.DefaultAttachmentManager.getAttachment(DefaultAttachmentManager.java:128)
        at com.atlassian.confluence.pages.DelegatorAttachmentManager.getAttachment(DelegatorAttachmentManager.java:105)
        at com.atlassian.confluence.pages.CachingAttachmentManager.getAttachment(CachingAttachmentManager.java:158)
        at sun.reflect.GeneratedMethodAccessor1230.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        at com.sun.proxy.$Proxy62.getAttachment(Unknown Source)
        at com.atlassian.confluence.servlet.download.AttachmentUrlParser.getAttachment(AttachmentUrlParser.java:51)
        at com.atlassian.confluence.importexport.resource.AttachmentDownloadResourceManager.getAttachment(AttachmentDownloadResourceManager.java:75)
        at com.atlassian.confluence.importexport.resource.AttachmentDownloadResourceManager.getResourceReader(AttachmentDownloadResourceManager.java:59)
        at com.atlassian.confluence.importexport.resource.DelegatorDownloadResourceManager.getResourceReader(DelegatorDownloadResourceManager.java:35)
        at com.atlassian.confluence.importexport.impl.AbstractRendererExporterImpl.exportResource(AbstractRendererExporterImpl.java:198)
        at com.atlassian.confluence.importexport.impl.HtmlExporter.doExportEntity(HtmlExporter.java:202)
        at com.atlassian.confluence.importexport.impl.HtmlExporter.exportPage(HtmlExporter.java:102)
        at com.atlassian.confluence.importexport.impl.HtmlExporter.exportSpace(HtmlExporter.java:112)
        at com.atlassian.confluence.importexport.impl.AbstractRendererExporterImpl.doExport(AbstractRendererExporterImpl.java:97)
        at com.atlassian.confluence.importexport.impl.HtmlExporter.doExport(HtmlExporter.java:46)
        at com.atlassian.confluence.importexport.DefaultImportExportManager.doExport(DefaultImportExportManager.java:139)
        at com.atlassian.confluence.importexport.DefaultImportExportManager.exportAs(DefaultImportExportManager.java:104)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:483)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
        at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
        at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
        at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
        at com.sun.proxy.$Proxy110.exportAs(Unknown Source)
        at com.atlassian.confluence.importexport.actions.ExportSpaceRunner.lambda$executeTask$0(ExportSpaceRunner.java:174)
        at com.atlassian.confluence.importexport.actions.ExportSpaceRunner$$Lambda$2595/1691379474.doInTransaction(Unknown Source)
        at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
        at com.atlassian.confluence.importexport.actions.ExportSpaceRunner.executeTask(ExportSpaceRunner.java:150)
        at com.atlassian.confluence.importexport.actions.ExportSpaceRunner.runTask(ExportSpaceRunner.java:97)
        at com.atlassian.progress.amq.TaskProcessor.accept(TaskProcessor.java:61)
        at com.atlassian.progress.amq.TaskProcessor.accept(TaskProcessor.java:21)
        at com.atlassian.progress.amq.ConcurrencyControlTaskProcessor.accept(ConcurrencyControlTaskProcessor.java:58)
        at com.atlassian.progress.amq.ConcurrencyControlTaskProcessor.accept(ConcurrencyControlTaskProcessor.java:18)
        at com.atlassian.progress.amq.ProgressAwareMessageRunner.processMessage(ProgressAwareMessageRunner.java:52)
        at com.atlassian.confluence.impl.messagequeue.VCacheMessageRunner.lambda$processMessage$0(VCacheMessageRunner.java:21)
        at com.atlassian.confluence.impl.messagequeue.VCacheMessageRunner$$Lambda$1322/1931356420.run(Unknown Source)
        at com.atlassian.confluence.vcache.VCacheRequestContextOperations.lambda$doInRequestContext$0(VCacheRequestContextOperations.java:49)
        at com.atlassian.confluence.vcache.VCacheRequestContextOperations$$Lambda$1319/170110890.perform(Unknown Source)
        at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:83)
        at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:73)
        at com.atlassian.confluence.vcache.VCacheRequestContextOperations.doInRequestContext(VCacheRequestContextOperations.java:48)
        at com.atlassian.confluence.impl.messagequeue.VCacheMessageRunner.processMessage(VCacheMessageRunner.java:20)
        at com.atlassian.messagequeue.internal.scheduler.SchedulerMessageRunnerService$AmqJobRunner.processNestedMessage(SchedulerMessageRunnerService.java:206)
        at com.atlassian.messagequeue.internal.scheduler.SchedulerMessageRunnerService$AmqJobRunner.runJob(SchedulerMessageRunnerService.java:163)
        at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:191)
        at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:164)
        at com.atlassian.scheduler.core.JobLauncher.launchJob(JobLauncher.java:135)
        at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:126)
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:480)
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:458)
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJobWithRecoveryGuard(CaesiumSchedulerService.java:463)
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:409)
        at com.atlassian.confluence.impl.schedule.caesium.UnicornSchedulerService.lambda$executeQueuedJob$0(UnicornSchedulerService.java:100)
        at com.atlassian.confluence.impl.schedule.caesium.UnicornSchedulerService$$Lambda$1318/236726961.run(Unknown Source)
        at com.atlassian.confluence.vcache.VCacheRequestContextOperations.lambda$doInRequestContext$0(VCacheRequestContextOperations.java:49)
        at com.atlassian.confluence.vcache.VCacheRequestContextOperations$$Lambda$1319/170110890.perform(Unknown Source)
        at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:83)
        at com.atlassian.confluence.impl.vcache.VCacheRequestContextManager.doInRequestContext(VCacheRequestContextManager.java:73)
        at com.atlassian.confluence.vcache.VCacheRequestContextOperations.doInRequestContext(VCacheRequestContextOperations.java:48)
        at com.atlassian.confluence.impl.schedule.caesium.UnicornSchedulerService.executeQueuedJob(UnicornSchedulerService.java:100)
        at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$$Lambda$1317/798053586.consume(Unknown Source)
        at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:65)
        at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:59)
        at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:34)
        at java.lang.Thread.run(Thread.java:745)

   Locked ownable synchronizers:
        - None

      Looking at the requests to the Media API, there's only around 30/minute.

      Confluence or the Media API should probably be batching the attachments rather than taking them one at a time.

              Assignee:
              Phil Wallach (Inactive)
              Reporter:
              Nick Mason
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

                Created:
                Updated:
                Resolved: