Details
Medium Description
Summary
Bitbucket Server stops responding.
Environment
- Firewall in place preventing a connection to the Marketplace
Steps to Reproduce
- Unknown
Expected Results
The application is always responding as expected.
Actual Results
No errors are logged. A thread dumps analysis shows a significant amount of long running threads similar to the following one:
"http-nio-7990-exec-1775" #8397 daemon prio=5 os_prio=0 tid=0x00007f20a01a9800 nid=0x5f68 waiting on condition [0x00007f2063953000] java.lang.Thread.State: WAITING (parking) at sun.misc.Unsafe.park(Native Method) - parking to wait for <0x00000000d87d6fe0> (a com.google.common.util.concurrent.AbstractFuture$Sync) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836) at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997) ... - locked <0x00000000dc06b428> (a org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
With the following thread being the one blocked from reading from https://marketplace.atlassian.com:
http-nio-7990-exec-1562" #6609 daemon prio=5 os_prio=0 tid=0x00007f209409f800 nid=0x34df runnable [0x00007f207585d000]
java.lang.Thread.State: RUNNABLE
at java.net.SocketInputStream.socketRead0(Native Method)
at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
at java.net.SocketInputStream.read(SocketInputStream.java:170)
at java.net.SocketInputStream.read(SocketInputStream.java:141)
at sun.security.ssl.InputRecord.readFully(InputRecord.java:465)
at sun.security.ssl.InputRecord.readV3Record(InputRecord.java:593)
at sun.security.ssl.InputRecord.read(InputRecord.java:532)
at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:973)
- locked <0x00000000d87dba78> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1375)
- locked <0x00000000d87dbb38> (a java.lang.Object)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1403)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1387)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:275)
at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:254)
at org.apache.http.impl.conn.HttpClientConnectionOperator.connect(HttpClientConnectionOperator.java:123)
at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:318)
at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:363)
at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:219)
at org.apache.http.impl.client.cache.CachingExec.revalidateCacheEntry(CachingExec.java:740)
at org.apache.http.impl.client.cache.CachingExec.revalidateCacheEntry(CachingExec.java:317)
at org.apache.http.impl.client.cache.CachingExec.handleCacheHit(CachingExec.java:287)
at org.apache.http.impl.client.cache.CachingExec.execute(CachingExec.java:264)
at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:195)
at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:86)
at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:108)
at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:184)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
at com.atlassian.marketplace.client.impl.CommonsHttpTransport$OperationsImpl.executeMethod(CommonsHttpTransport.java:376)
at com.atlassian.marketplace.client.impl.CommonsHttpTransport$OperationsImpl.get(CommonsHttpTransport.java:298)
at com.atlassian.marketplace.client.impl.CommonsHttpTransport.get(CommonsHttpTransport.java:112)
at com.atlassian.marketplace.client.impl.ApiHelper.getEntityInternal(ApiHelper.java:113)
at com.atlassian.marketplace.client.impl.ApiHelper.getEntity(ApiHelper.java:100)
at com.atlassian.marketplace.client.impl.DefaultMarketplaceClient.getRoot(DefaultMarketplaceClient.java:156)
at com.atlassian.marketplace.client.impl.DefaultMarketplaceClient.applications(DefaultMarketplaceClient.java:121)
at com.atlassian.upm.pac.PacClientImpl.fetchMpacAppInfo(PacClientImpl.java:299)
at com.atlassian.upm.pac.PacClientImpl.fetchMpacAppInfoForLocalBuild(PacClientImpl.java:323)
at com.atlassian.upm.pac.PacClientImpl.access$700(PacClientImpl.java:119)
at com.atlassian.upm.pac.PacClientImpl$6$1.apply(PacClientImpl.java:285)
at com.atlassian.upm.pac.PacClientImpl$6$1.apply(PacClientImpl.java:281)
at com.atlassian.upm.api.util.Option$Some.fold(Option.java:314)
at com.atlassian.upm.api.util.Option.flatMap(Option.java:117)
at com.atlassian.upm.pac.PacClientImpl$6.get(PacClientImpl.java:280)
at com.atlassian.upm.pac.PacClientImpl$6.get(PacClientImpl.java:270)
at com.atlassian.cache.compat.guava11.memory.MemoryCacheFactory$1$1.load(MemoryCacheFactory.java:50)
at com.atlassian.cache.compat.guava11.memory.MemoryCacheFactory$1$1.load(MemoryCacheFactory.java:46)
at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3527)
at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2319)
at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2282)
- locked <0x00000000d87d7078> (a com.google.common.cache.LocalCache$StrongAccessWriteEntry)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2197)
at com.google.common.cache.LocalCache.get(LocalCache.java:3937)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3941)
at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4824)
at com.atlassian.cache.compat.guava11.memory.MemoryCachedReference.get(MemoryCachedReference.java:37)
at com.atlassian.upm.pac.PacClientImpl.mpacApp(PacClientImpl.java:355)
at com.atlassian.upm.pac.PacClientImpl.unknownAppVersion(PacClientImpl.java:359)
at com.atlassian.upm.pac.PacClientImpl.isUnknownProductVersion(PacClientImpl.java:350)
at com.atlassian.upm.permission.UpmVisibilityImpl.isKnownOrDevelopmentVersion(UpmVisibilityImpl.java:72)
at com.atlassian.upm.permission.UpmVisibilityImpl.isInstallVisible(UpmVisibilityImpl.java:45)
at com.atlassian.upm.conditions.IsPermittedToView.shouldDisplay(IsPermittedToView.java:61)
at com.atlassian.plugin.web.DefaultWebInterfaceManager.filterFragmentsByCondition(DefaultWebInterfaceManager.java:154)
at com.atlassian.plugin.web.DefaultWebInterfaceManager.getDisplayableItems(DefaultWebInterfaceManager.java:107)
at com.atlassian.plugin.web.DefaultWebInterfaceManager.getDisplayableWebItems(DefaultWebInterfaceManager.java:215)
at com.atlassian.stash.internal.web.fragments.WebFragmentSupport.getWebItems(WebFragmentSupport.java:61)
at com.atlassian.stash.internal.web.fragments.WebFragmentSupport$$FastClassBySpringCGLIB$$a614745b.invoke(<generated>)
Notes
Workaround
Disable the connection to the Universal Plugin Manager (UPM) / Atlassian Marketplace, by adding the following to the <$Bitbucket Installation directory>/bin/start-webapp.sh file:
-Dupm.pac.disable=true
A restart is required for this change to take effect.
Note that this line may be commented on the servers. If that's the case the line will start with # and we will need to remove that. If there are already some parameters set, please add the new one.