Creating or editing a page in Confluence when applinks are unreachable can intermittently cause the editor to hang

XMLWordPrintable

    • 17
    • Severity 3 - Minor
    • 2

      Issue Summary

      When a user creates or edits a page in Confluence, a call is made to /rest/jiraanywhere/1.0/servers to pre-populate a list of Jira servers that are available. The first time this URL is requested after startup, Confluence will attempt to retrieve server details from all Jira servers with a configured application link within Confluence. Those results are then cached for 4 hours, then the next call to /rest/jiraanywhere/1.0/servers will refresh those cached values.

      Typical output with one application link for Jira looks like:

      [{"id":"a11fa272-0a98-37e5-86b3-e563cce10260","name":"Jira","selected":true,"url":"http://pizza.atlassian.com:8080/jira","buildNumber":9223372036854775807}]

      Flushing caches via the administration screen in Confluence seemingly has no impact on the 4 hour cache TTL for this request.

      Delay Impact 
      If a Jira server is unreachable, the first page edit/create operation after every 4 hours will pause for at least 10 seconds, effectively hanging the Confluence editor for the same amount of time. If 2 Jira servers are unreachable (2x10sec), the delay will be at least 20 seconds.

      If an outbound proxy is in use, the delays can be much more pronounced, to the point where the editor fails to load completely. If this occurs, the only available workaround in this instance is to refresh the page.

      Example timeline:

      Time Event
      09:00:00 Confluence is started
      09:05:00 User creates a page, which triggers the call to /rest/jiraanywhere/1.0/servers
      09:30:00 Another user creates a page, which uses the cached data from the 09:05:00 call
      12:55:00 Another user edits a page, which also uses the cached data from the 09:05:00 call
      13:10:00 A user creates a page, but 4 hours has elapsed since 09:05:00, so the previously cached data from the earlier /rest/jiraanywhere/1.0/servers call has expired, so Confluence attempts to contact all Jira servers again to repopulate the cache.

       In the above example, the editor would hang at 09:05:00 and 13:10:00.

      Steps to Reproduce

      1. Start Confluence
      2. Create an application link to a Jira instance
      3. Shutdown Confluence
      4. Modify the applink via the database, changing the display and rpc url's to an unreachable address that times out and doesn't just immediately reject the connection (eg. 203.56.8.1) 
        From the BANDANA table:

   1048582 | _GLOBAL        | applinks.admin.a11fa272-0a98-37e5-86b3-e563cce10260.display.url                           | <string>http://pizza.atlassian.com:8080/jira</string>
   1048583 | _GLOBAL        | applinks.admin.a11fa272-0a98-37e5-86b3-e563cce10260.rpc.url                               | <string>http://pizza.atlassian.com:8080/jira</string> 

update bandana set bandanavalue='<string>http://203.56.8.1:8080/jira</string>' where bandanaid=1048582;
update bandana set bandanavalue='<string>http://203.56.8.1:8080/jira</string>' where bandanaid=1048583;
      5. Start Confluence
      6. Create or edit a page

      For even longer delays, configure an outbound proxy (like datadog/squid) and then configure setenv.sh with the proxy settings:

      CATALINA_OPTS="-Dhttps.proxyHost=127.0.0.1 -Dhttps.proxyPort=3128  ${CATALINA_OPTS}"

      Expected Results

      The page editor should load if an applink server is unreachable.

      Actual Results

      Without an outbound proxy, the editor will hang for 10 seconds for each applink that is unreachable. If there is an outbound proxy, the problem becomes much worse and then editor will often hang indefinitely with the following error displayed on the UI:

      If the delay exceeds 60sec (such as with an outbound proxy configured), the below exception is thrown in the atlassian-confluence.log file for the stuck thread:

      2021-05-24 13:54:54,844 WARN [http-nio-8090-exec-9] [confluence.util.profiling.DefaultActivityMonitor] close Exceeded the threshold of 60000 ms: ActivitySnapshot{startTime=1621828386267, threadId=241, threadName='http-nio-8090-exec-9', userId='user2', type='web-request', summary='/rest/jiraanywhere/1.0/servers?_=1621828386255'}
 -- referer: http://localhost:8090/pages/createpage.action?spaceKey=ST | url: /rest/jiraanywhere/1.0/servers | traceId: 5668110629b8a971 | userName: user2

      Also with a StuckThreadDetectionValve set in server.xml, the catalina.out file will display:

      24-May-2021 13:54:10.421 WARNING [Catalina-utility-3] org.apache.catalina.valves.StuckThreadDetectionValve.notifyStuckThreadDetected Thread [http-nio-8090-exec-9] (id=[241]) has been active for [64,155] milliseconds (since [5/24/21 1:53 PM]) to serve the same request for [http://localhost:8090/rest/jiraanywhere/1.0/servers?_=1621828386255] and may be stuck (configured threshold for this StuckThreadDetectionValve is [60] seconds). There is/are [1] thread(s) in total that are monitored by this Valve and may be stuck.
 java.lang.Throwable
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at org.apache.http.impl.io.SessionInputBufferImpl.streamRead(SessionInputBufferImpl.java:137)
        at org.apache.http.impl.io.SessionInputBufferImpl.fillBuffer(SessionInputBufferImpl.java:153)
        at org.apache.http.impl.io.SessionInputBufferImpl.readLine(SessionInputBufferImpl.java:282)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:138)
        at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:56)
        at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:259)
        at org.apache.http.impl.DefaultBHttpClientConnection.receiveResponseHeader(DefaultBHttpClientConnection.java:163)
        at org.apache.http.impl.conn.CPoolProxy.receiveResponseHeader(CPoolProxy.java:165)
        at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:273)
        at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:125)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:272)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:111)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at com.atlassian.sal.core.net.HttpClientRequest.executeAndReturn(HttpClientRequest.java:105)
        at com.atlassian.sal.core.net.HttpClientRequest.execute(HttpClientRequest.java:75)
        at com.atlassian.plugins.rest.module.jersey.JerseyRequest.execute(JerseyRequest.java:125)
        at com.atlassian.applinks.core.auth.ApplicationLinkRequestAdaptor.execute(ApplicationLinkRequestAdaptor.java:42)
        at com.atlassian.confluence.extra.jira.applink.DefaultJiraConnectorManager.getServerBuildNumber(DefaultJiraConnectorManager.java:84)
        at com.atlassian.confluence.extra.jira.applink.DefaultJiraConnectorManager.createJiraServerBean(DefaultJiraConnectorManager.java:77)
        at com.atlassian.confluence.extra.jira.applink.DefaultJiraConnectorManager.access$000(DefaultJiraConnectorManager.java:24)
        at com.atlassian.confluence.extra.jira.applink.DefaultJiraConnectorManager$1.load(DefaultJiraConnectorManager.java:119)
        at com.atlassian.confluence.extra.jira.applink.DefaultJiraConnectorManager$1.load(DefaultJiraConnectorManager.java:116)
        at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
        at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
        at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
        at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4830)
        at com.atlassian.confluence.extra.jira.applink.DefaultJiraConnectorManager.getInternalJiraServer(DefaultJiraConnectorManager.java:100)
        at com.atlassian.confluence.extra.jira.applink.DefaultJiraConnectorManager.getJiraServers(DefaultJiraConnectorManager.java:47)
        at com.atlassian.confluence.plugins.jira.rest.JiraConnectorStateResource.getJiraServers(JiraConnectorStateResource.java:38)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$ResponseOutInvoker$1.invoke(DispatchProviderHelper.java:192)
        at com.atlassian.plugins.rest.common.interceptor.impl.DispatchProviderHelper$1.intercept(DispatchProviderHelper.java:83)
<...snip...>

      Workaround

      Remove any unreachable applinks.

        1. jiraanywhere-editor-loading.png
          269 kB
          Malcolm Ninnes

            Assignee:
            Arpan Agrawal
            Reporter:
            Malcolm Ninnes
            Votes:
            8 Vote for this issue
            Watchers:
            21 Start watching this issue

              Created:
              Updated:
              Resolved: