Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-16596

Diffing page content via rss causes high load

XMLWordPrintable

      As mentioned in CONF-17499, some pages take a long time to diff. In Confluence instances where RSS is used widely, the result of such a page being updated can result in a great amount of requests to diff that page. Thread dumps will have many entries from createrssfeed.action actions like this one:

      "TP-Processor23" daemon prio=10 tid=0x088a9000 nid=0x6fbe runnable [0x5f9b7000..0x5f9bada0]
  java.lang.Thread.State: RUNNABLE
	at org.apache.commons.jrcs.diff.myers.PathNode.previousSnake(PathNode.java:123)
	at org.apache.commons.jrcs.diff.myers.DiffNode.<init>(DiffNode.java:45)
	at org.apache.commons.jrcs.diff.myers.MyersDiff.buildPath(MyersDiff.java:153)
	at org.apache.commons.jrcs.diff.myers.MyersDiff.diff(MyersDiff.java:93)
	at org.apache.commons.jrcs.diff.Diff.diff(Diff.java:197)
	at org.apache.commons.jrcs.diff.Diff.diff(Diff.java:185)
	at org.apache.commons.jrcs.diff.Diff.diff(Diff.java:165)
	at com.atlassian.confluence.diff.DeltaRanker.getChangeRanking(DeltaRanker.java:27)
	at com.atlassian.confluence.diff.LineRanker.getIndexOfSmallestChange(LineRanker.java:30)
	at com.atlassian.confluence.diff.LineLevelDiffer.getChunksForChangedLines(LineLevelDiffer.java:214)
	at com.atlassian.confluence.diff.LineLevelDiffer.diff(LineLevelDiffer.java:91)
	at com.atlassian.confluence.diff.LineLevelDiffer.diff(LineLevelDiffer.java:346)
	at com.atlassian.confluence.diff.LineLevelDiffer.diff(LineLevelDiffer.java:333)
	at com.atlassian.confluence.diff.ConfluenceDiff.<init>(ConfluenceDiff.java:22)
	at com.atlassian.confluence.diff.ConfluenceDiff.<init>(ConfluenceDiff.java:29)
	at com.atlassian.confluence.rss.PageRenderSupport.getDiff(PageRenderSupport.java:98)
	at com.atlassian.confluence.rss.PageRenderSupport.getRenderedContent(PageRenderSupport.java:64)
	at com.atlassian.confluence.rss.DefaultFeedBuilder.createFeed(DefaultFeedBuilder.java:103)
	at sun.reflect.GeneratedMethodAccessor628.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
	at java.lang.reflect.Method.invoke(Method.java:597)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:304)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:182)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:149)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:106)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:171)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
	at $Proxy62.createFeed(Unknown Source)
	at com.atlassian.confluence.spaces.actions.CreateRssFeedAction.getSyndFeed(CreateRssFeedAction.java:51)

      To address this in Confluence 3.1, a limit was introduced to the maximum number of concurrent diffs. When that number is exceeded, additional attempts by RSS feeds to create diffs are ignored and logged. (The RSS requests succeed, they are just missing diffs).

      Option configuration

      The size of the threadpool defaults to 20, and can be configured with the system property "confluence.diff.pool.size".
      The default timeout is 1000 ms, and can be configured with the system property "confluence.diff.timeout".

        1. CONF-16596.patch
          3 kB

          Form Name

            [CONFSERVER-16596] Diffing page content via rss causes high load

            Don Willis added a comment -

            Updating fix version from 3.2->3.1, since that is the version in which the mitigation "fixed" the problem.

            Don Willis added a comment - Updating fix version from 3.2->3.1, since that is the version in which the mitigation "fixed" the problem.

            Per Fragemann [Atlassian] added a comment -

            I also believe the mitigation is fine for now. Tentatively scheduling for 3.2.

            Per Fragemann [Atlassian] added a comment - I also believe the mitigation is fine for now. Tentatively scheduling for 3.2.

            Don Willis added a comment -

            We decided that the current mitigation should be sufficient for 3.1, and that there is no particular pain for diffing through RSS. My current suspicion is that diffing is quite CPU intensive and we have a lot of RSS requests, so we should look into improving their performance. An issue has been created to add performance tests for diffs in RSS so we can look into improving their performance.

            The 3.2 ER team should look into this issue quickly, since it may greatly improve performance on EAC and make us all happier.

            Don Willis added a comment - We decided that the current mitigation should be sufficient for 3.1, and that there is no particular pain for diffing through RSS. My current suspicion is that diffing is quite CPU intensive and we have a lot of RSS requests, so we should look into improving their performance. An issue has been created to add performance tests for diffs in RSS so we can look into improving their performance. The 3.2 ER team should look into this issue quickly, since it may greatly improve performance on EAC and make us all happier.

            Per Fragemann [Atlassian] added a comment -

            If this can't be closed, we need to pull the plug on the feature.

            Per Fragemann [Atlassian] added a comment - If this can't be closed, we need to pull the plug on the feature.

            Don Willis added a comment -

            This shouldn't have been closed. The bug still exists, we've just introduced a mitigating factor that should help us diagnose the problem.

            Don Willis added a comment - This shouldn't have been closed. The bug still exists, we've just introduced a mitigating factor that should help us diagnose the problem.

            Matt Ryall added a comment -

            I talked to Jed about making some changes. Rolling back to tech review until these are in.

            Matt Ryall added a comment - I talked to Jed about making some changes. Rolling back to tech review until these are in.

            Matt Ryall added a comment -

            We've changed Confluence so all diff operations run on a separate thread and the request thread will wait up to 1 second for any single diff to complete. The timeout is configurable via the 'confluence.diff.timeout' system property, which specifies the number of milliseconds to wait (default: 1000). If the diff doesn't complete within that timeframe, an error message will be logged and a note shown to the user in the web interface, email or RSS feed.

            We should still investigate why diffing some content takes a really long time, so I've raised CONF-17499 to track this.

            Don and I paired on this. No need for further technical review.

            Matt Ryall added a comment - We've changed Confluence so all diff operations run on a separate thread and the request thread will wait up to 1 second for any single diff to complete. The timeout is configurable via the 'confluence.diff.timeout' system property, which specifies the number of milliseconds to wait (default: 1000). If the diff doesn't complete within that timeframe, an error message will be logged and a note shown to the user in the web interface, email or RSS feed. We should still investigate why diffing some content takes a really long time, so I've raised CONF-17499 to track this. Don and I paired on this. No need for further technical review.

            Per Fragemann [Atlassian] added a comment -

            This one sounds serious. Given that David is buried under bugs in the Restrictions dialog and with smart fields, and given that Matt has worked a lot with 3.0 performance, I am assigning this to Matt for making a decision. If no simple solution/workaround/queue/setting/miracle can be found, we need to revert the functionality and postpone it.

            Per Fragemann [Atlassian] added a comment - This one sounds serious. Given that David is buried under bugs in the Restrictions dialog and with smart fields, and given that Matt has worked a lot with 3.0 performance, I am assigning this to Matt for making a decision. If no simple solution/workaround/queue/setting/miracle can be found, we need to revert the functionality and postpone it.

            David Taylor (Inactive) added a comment -

            Try that feed URL without showDiff=true and it should work.

            David Taylor (Inactive) added a comment - Try that feed URL without showDiff=true and it should work.

            PdZ (Inactive) added a comment -

            No RSS feed creation on EAC makes keeping up on Atlassian news somewhat difficult:

            PdZ (Inactive) added a comment - No RSS feed creation on EAC makes keeping up on Atlassian news somewhat difficult:

              Unassigned Unassigned
              dcheney David Cheney (Inactive)
              Affected customers:
              1 This affects my team
              Watchers:
              6 Start watching this issue

                Created:
                Updated:
                Resolved: