Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-13875

Confluence Daily Mail Notification eats up CPU performance

    XMLWordPrintable

Details

    Description

      A customer (https://support.atlassian.com/browse/CSP-24847) found that Confluence Daily notification is consuming up to 100% of the CPU (1 of 4 CPUs) performance. This performance issue would run up to several hours, causing slow overall Confluence performance.

      His instance is quite large and has over 5000 updates everyday.

      Using Matt Ryall's script, I was able to imitate this behaviour in a local instance and noticed a 100% CPU spike (1 of 8 CPUs) - it goes up and down but on average it's between 80% - 90%.

      I created 6000 pages in a single space using the script and set one user to receive the daily update which was sent to a local mail server.

      I have set the DailyReportJob to run at 3PM and received the email update over one hour later. During the period the most active thread was this thread:

      "DefaultQuartzScheduler_Worker-1" prio=5 tid=0x01064920 nid=0xa86a00 runnable [0xb1295000..0xb1296d90]
	at java.lang.System.identityHashCode(Native Method)
	at net.sf.hibernate.util.IdentityMap$IdentityKey.hashCode(IdentityMap.java:103)
	at java.util.HashMap.get(HashMap.java:343)
	at org.apache.commons.collections.SequencedHashMap.get(SequencedHashMap.java:288)
	at net.sf.hibernate.util.IdentityMap.get(IdentityMap.java:126)
	at net.sf.hibernate.impl.SessionImpl.getEntry(SessionImpl.java:711)
	at net.sf.hibernate.impl.SessionImpl.getEntityIdentifier(SessionImpl.java:2751)
	at net.sf.hibernate.impl.SessionImpl.updateReachableCollection(SessionImpl.java:2913)
	at net.sf.hibernate.impl.FlushVisitor.processCollection(FlushVisitor.java:32)
	at net.sf.hibernate.impl.AbstractVisitor.processValue(AbstractVisitor.java:69)
	at net.sf.hibernate.impl.AbstractVisitor.processValues(AbstractVisitor.java:36)
	at net.sf.hibernate.impl.SessionImpl.flushEntity(SessionImpl.java:2620)
	at net.sf.hibernate.impl.SessionImpl.flushEntities(SessionImpl.java:2486)
	at net.sf.hibernate.impl.SessionImpl.flushEverything(SessionImpl.java:2281)
	at net.sf.hibernate.impl.SessionImpl.flush(SessionImpl.java:2260)
	at net.sf.hibernate.transaction.JDBCTransaction.commit(JDBCTransaction.java:61)
	at org.springframework.orm.hibernate.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:510)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.processCommit(AbstractPlatformTransactionManager.java:662)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:632)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.commitTransactionAfterReturning(TransactionAspectSupport.java:314)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:116)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at $Proxy23.getUser(Unknown Source)
	at com.atlassian.confluence.core.ConfluenceActionSupport.getUser(ConfluenceActionSupport.java:301)
	at com.atlassian.confluence.core.ConfluenceActionSupport.getUserFullName(ConfluenceActionSupport.java:309)
	at sun.reflect.GeneratedMethodAccessor279.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:585)
	at org.apache.velocity.util.introspection.UberspectImpl$VelMethodImpl.invoke(UberspectImpl.java:295)
	at com.atlassian.confluence.velocity.introspection.UnboxingMethod.invoke(UnboxingMethod.java:29)
	at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:245)
	at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:203)
	at org.apache.velocity.runtime.parser.node.ASTReference.value(ASTReference.java:419)
	at org.apache.velocity.runtime.parser.node.ASTMethod.execute(ASTMethod.java:147)
	at org.apache.velocity.runtime.parser.node.ASTReference.execute(ASTReference.java:203)
	at org.apache.velocity.runtime.parser.node.ASTReference.render(ASTReference.java:294)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:74)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:88)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:74)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:88)
	at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:318)
	at org.apache.velocity.runtime.directive.VelocimacroProxy.render(VelocimacroProxy.java:194)
	at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:170)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:74)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:88)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:74)
	at org.apache.velocity.runtime.directive.Foreach.render(Foreach.java:448)
	at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:170)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:74)
	at org.apache.velocity.runtime.parser.node.ASTIfStatement.render(ASTIfStatement.java:88)
	at org.apache.velocity.runtime.parser.node.ASTBlock.render(ASTBlock.java:74)
	at org.apache.velocity.runtime.directive.Foreach.render(Foreach.java:448)
	at org.apache.velocity.runtime.parser.node.ASTDirective.render(ASTDirective.java:170)
	at org.apache.velocity.runtime.parser.node.SimpleNode.render(SimpleNode.java:318)
	at org.apache.velocity.Template.merge(Template.java:254)
	at com.atlassian.confluence.util.velocity.VelocityUtils.getRenderedTemplateWithoutSwallowingErrors(VelocityUtils.java:56)
	at com.atlassian.confluence.util.velocity.VelocityUtils.getRenderedTemplate(VelocityUtils.java:34)
	at com.atlassian.confluence.util.velocity.VelocityUtils.getRenderedTemplate(VelocityUtils.java:27)
	at com.atlassian.confluence.mail.template.AbstractMailNotificationQueueItem.getRenderedContent(AbstractMailNotificationQueueItem.java:236)
	at com.atlassian.confluence.mail.template.PreRenderedMailNotificationQueueItem.render(PreRenderedMailNotificationQueueItem.java:74)
	at com.atlassian.confluence.mail.jobs.DailyReportJob.doExecute(DailyReportJob.java:81)
	at com.atlassian.quartz.jobs.AbstractJob.executeInternal(AbstractJob.java:87)
	at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
	at com.atlassian.confluence.setup.quartz.DelegatingClusterAwareQuartzJobBean.executeJob(DelegatingClusterAwareQuartzJobBean.java:16)
	at com.atlassian.confluence.setup.quartz.AbstractClusterAwareQuartzJobBean.surroundJobExecutionWithLogging(AbstractClusterAwareQuartzJobBean.java:64)
	at com.atlassian.confluence.setup.quartz.AbstractClusterAwareQuartzJobBean.executeInternal(AbstractClusterAwareQuartzJobBean.java:47)
	at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:203)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520)

      Here is the full thread dumps file

      PS: The final thread dump was taken one hour later, when the DefaultQuartzScheduler_Worker-1 has finished its job.

      Attachments

        1. blast-pages.py
          0.6 kB
          Roy Hartono [Atlassian]
        2. threadDumps.txt
          204 kB
          Roy Hartono [Atlassian]

        Issue Links

          causes

          Bug - A problem which impairs or prevents the functions of the product. CONFSERVER-21645 Daily report stops functioning when user is removed directly from LDAP server

          • Low - Low priority issues
          • Closed
          is duplicated by

          Suggestion - CONFSERVER-14035 at com.atlassian.confluence.mail.jobs.DailyReportJob.doExecute performs poorly with a large number of spaces/pages

          • Closed

          Activity

            People

              cpetchell Petch (Inactive)
              rhartono Roy Hartono [Atlassian]
              Votes:
              4 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: