Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-58927

If the Marketplace is unreachable, UPM causes Confluence 7.0.1 to hang for 30sec per UI click

      Issue Summary

      If the Marketplace is unreachable, UPM 4.0.6 causes Confluence 7.0.1 to hang for 30sec per UI click.

      Environment

      • UPM 4.0.6 in Confluence 7.0.1
      • marketplace.atlassian.com non-responsive. This needs to be a timeout, not a straight connection refusal by a firewall.

      Steps to Reproduce

      1. Add entry in /etc/hosts to simulate marketplace being unreachable:
         203.56.8.1 marketplace.atlassian.com
        
      1. Click anything in Confluence and get ready to wait, and wait....

      Expected Results

      The Confluence UI should remain responsive even if Marketplace is unreachable.

      Actual Results

      The result of each click in the UI (eg. clicking on Spaces, or trying to load the Confluence admin screen) takes approximately 30 seconds, giving the impression that Confluence performance is very poor.

      Notes

      If the IP that marketplace.atlassian.com resolves to refuses or accepts the connection immediately, there is no problem - it's only when the resolved IP fails to connect and eventually times out.

      Confluence thread dump on a hanging thread:

      at java.net.PlainSocketImpl.socketConnect(Native Method)
      at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
      - locked [0x00000007bdeaf1e8] (a java.net.SocksSocketImpl)
      at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
      at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
      at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
      at java.net.Socket.connect(Socket.java:589) 
      at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339)
      at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
      at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
      at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:394)
      at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
      at org.apache.http.impl.client.cache.CachingExec.callBackend(CachingExec.java:592)
      at org.apache.http.impl.client.cache.CachingExec.handleCacheMiss(CachingExec.java:356)
      at org.apache.http.impl.client.cache.CachingExec.execute(CachingExec.java:275) 
      at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
      at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
      at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
      at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
      at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56)
      at com.atlassian.marketplace.client.impl.CommonsHttpTransport$OperationsImpl.executeMethod(CommonsHttpTransport.java:389)
      at com.atlassian.marketplace.client.impl.CommonsHttpTransport$OperationsImpl.get(CommonsHttpTransport.java:303)
      at com.atlassian.marketplace.client.impl.CommonsHttpTransport.get(CommonsHttpTransport.java:115)
      at com.atlassian.marketplace.client.impl.ApiHelper.getEntityInternal(ApiHelper.java:114)
      at com.atlassian.marketplace.client.impl.ApiHelper.getEntity(ApiHelper.java:101)
      at com.atlassian.marketplace.client.impl.DefaultMarketplaceClient.getRoot(DefaultMarketplaceClient.java:156)
      at com.atlassian.marketplace.client.impl.DefaultMarketplaceClient.applications(DefaultMarketplaceClient.java:121) 
      at com.atlassian.upm.pac.PacClientImpl.fetchMpacAppInfo(PacClientImpl.java:228)
      at com.atlassian.upm.pac.PacClientImpl.fetchMpacAppInfoForLocalBuild(PacClientImpl.java:252)
      at com.atlassian.upm.pac.PacClientImpl.lambda$createMpacAppRef$1(PacClientImpl.java:217)
      at com.atlassian.upm.pac.PacClientImpl$$Lambda$964/1871514796.get(Unknown Source)
      at com.atlassian.confluence.cache.DeferredCachedReference.get(DeferredCachedReference.java:61)
      at com.atlassian.confluence.cache.TransactionalCacheFactory$TransactionAwareCachedReference.get(TransactionalCacheFactory.java:396)
      at com.atlassian.upm.pac.PacClientImpl.mpacApp(PacClientImpl.java:277)
      at com.atlassian.upm.pac.PacClientImpl.unknownAppVersion(PacClientImpl.java:281)
      at com.atlassian.upm.pac.PacClientImpl.isUnknownProductVersion(PacClientImpl.java:272) 
      at com.atlassian.upm.permission.UpmVisibilityImpl.isKnownOrDevelopmentVersion(UpmVisibilityImpl.java:72)
      at com.atlassian.upm.permission.UpmVisibilityImpl.isInstallVisible(UpmVisibilityImpl.java:45)
      at com.atlassian.upm.conditions.IsPermittedToView.shouldDisplay(IsPermittedToView.java:61)
      at com.atlassian.plugin.web.DefaultWebInterfaceManager.filterFragmentsByCondition(DefaultWebInterfaceManager.java:160)
      at com.atlassian.plugin.web.DefaultWebInterfaceManager.getDisplayableItems(DefaultWebInterfaceManager.java:113)
      at com.atlassian.confluence.plugin.descriptor.web.ConfluenceWebInterfaceManager.getDisplayableItems(ConfluenceWebInterfaceManager.java:78)
      at com.atlassian.confluence.web.ConfluenceWebMenuManager.getSectionItems(ConfluenceWebMenuManager.java:48)
      at com.atlassian.confluence.web.ConfluenceWebMenuManager.getMenu(ConfluenceWebMenuManager.java:32)

      Stuck Thread

      catalina.out may report stuck threads that are related to the Marketplace connection:

      		at com.atlassian.upm.pac.PacClientImpl.mpacApp(PacClientImpl.java:277)
      		at com.atlassian.upm.pac.PacClientImpl.unknownAppVersion(PacClientImpl.java:281)
      		at com.atlassian.upm.pac.PacClientImpl.isUnknownProductVersion(PacClientImpl.java:272)
      		at com.atlassian.upm.permission.UpmVisibilityImpl.isKnownOrDevelopmentVersion(UpmVisibilityImpl.java:72)
      		at com.atlassian.upm.permission.UpmVisibilityImpl.isInstallVisible(UpmVisibilityImpl.java:45)
      		at com.atlassian.upm.conditions.IsPermittedToView.shouldDisplay(IsPermittedToView.java:61)
      

      Workaround

      In Confluence, disable connectivity to Marketplace:

      1. Navigate to the cog icon > General Configuration > Manage Apps
      2. Click on Settings at the bottom of the page
      3. Uncheck Connect to Atlassian Marketplace and click Apply

      Workaround via REST API

      If Confluence is completely inaccessible we can use the following REST API to disable connectivity to Marketplace.

      curl -u username:password -X PUT <Confluence Base URL>/rest/plugins/1.0/settings -H 'Content-Type: application/vnd.atl.plugins+json' --data-binary '{"settings":[{"key":"pacDisabled","value":true,"requiresRefresh":true,"defaultCheckedValue":true,"readOnly":false}]}'
      

      This ticket is also tracked under UPM-6053

            [CONFSERVER-58927] If the Marketplace is unreachable, UPM causes Confluence 7.0.1 to hang for 30sec per UI click

            Zac Xu added a comment -

            A fix for this issue is available to Server and Data Center customers in Confluence 7.1.0
            Upgrade now or check out the Release Notes to see what other issues are resolved.

            Zac Xu added a comment - A fix for this issue is available to Server and Data Center customers in Confluence 7.1.0 Upgrade now or check out the Release Notes to see what other issues are resolved.

            Zac Xu added a comment -

            A fix for this issue is available to Server and Data Center customers in Confluence 7.0.4
            Upgrade now or check out the Release Notes to see what other issues are resolved.

            Zac Xu added a comment - A fix for this issue is available to Server and Data Center customers in Confluence 7.0.4 Upgrade now or check out the Release Notes to see what other issues are resolved.

            ingo.feulner I am unaware of any similar behaviour with emails, can you please share a thread dump if possible , it sounds to me like another issue altogether , looking into it this week

            Hasnae (Inactive) added a comment - ingo.feulner  I am unaware of any similar behaviour with emails, can you please share a thread dump if possible , it sounds to me like another issue altogether , looking into it this week

            Was facing this issue in 7.0.3.

            Suggested workaround seems fixed the issue temporarily.

            Deleted Account (Inactive) added a comment - Was facing this issue in 7.0.3. Suggested workaround seems fixed the issue temporarily.

            We see a similar behavior if emails can't be sent, because of unavailability of the configured mail server. Could there be a correlation? Had to restart Confluence server several times a day, as threads where blocked when trying to send emails (notifications/ suggested content).

            Ingo Feulner added a comment - We see a similar behavior if emails can't be sent, because of unavailability of the configured mail server. Could there be a correlation? Had to restart Confluence server several times a day, as threads where blocked when trying to send emails (notifications/ suggested content).

            MikeN added a comment -

            Please note: this issue is not fixed in 7.0.3 like Zac says.

            Apparently it is 'waiting for release' in 7.0.4.

            This kills our Confluence install every couple of days or so. I do not really understand why this was not fixed with the highest priority...

            MikeN added a comment - Please note: this issue is not fixed in 7.0.3 like Zac says. Apparently it is 'waiting for release' in 7.0.4. This kills our Confluence install every couple of days or so. I do not really understand why this was not fixed with the highest priority...

            Wow, this is the reason what I got the white screen and 504 error.

            ELB in AWS gives 504 when the site is not reachable for 30 seconds, so I thought 7.0.3 version of confluence is broken. This version throws so many stuck thread error and gives white screen about 30 seconds, and finally gives the error.

            However, if I did not use the ELB then it would work as the description says. I just want to let you know that the situation is worse with the environments in AWS.

            Bespin Global Admin added a comment - Wow, this is the reason what I got the white screen and 504 error. ELB in AWS gives 504 when the site is not reachable for 30 seconds, so I thought 7.0.3 version of confluence is broken. This version throws so many stuck thread error and gives white screen about 30 seconds, and finally gives the error. However, if I did not use the ELB then it would work as the description says. I just want to let you know that the situation is worse with the environments in AWS.

            Zac Xu added a comment - - edited

            A fix for this issue is available to Server and Data Center customers in Confluence 7.0.3
            Upgrade now or check out the Release Notes to see what other issues are resolved.

            Zac Xu added a comment - - edited A fix for this issue is available to Server and Data Center customers in Confluence 7.0.3 Upgrade now or check out the Release Notes to see what other issues are resolved.

            Fixing this properly depends on a new feature in atlassian-cache. I'll ensure this is safe to land in Confluence 7.0.x.

            Richard Atkins added a comment - Fixing this properly depends on a new feature in atlassian-cache. I'll ensure this is safe to land in Confluence 7.0.x.

            Requires a fix for UPM-6053 in UPM

            Richard Atkins added a comment - Requires a fix for UPM-6053 in UPM

              hrehioui Hasnae (Inactive)
              mninnes@atlassian.com Malcolm Ninnes
              Affected customers:
              11 This affects my team
              Watchers:
              53 Start watching this issue

                Created:
                Updated:
                Resolved: