Loading...

Details

Type: Bug
Resolution: Fixed
Priority: High
Fix Version/s: 5.11.0
Affects Version/s: 5.6.2
Component/s: JIRA integration
Labels:
- support

Bug Fix Policy:
View Atlassian Server bug fix policy

Description

Summary

Bamboo hang or unresponsive if there is an Application Link with JIRA

Problem

JIRA dashboard with unauthenticated Bamboo gadgets will make a call to /plugins/servlet/oauth/request-token generating a new entry (token for authentication request) in Bandana table for every request to every gadget.

In case of JIRA dashboard with 5 gadgets refreshed every 15 minutes for 5 users it gives 100 new records per hour. OAuth library is aware of this issue and contains ExpiredSessionRemover which is run every 8 hours and should clean expired tokens. We are verifying right now if this job is being run in Bamboo and if it actually clears expired records. This ticket will be updated based on our findings and recent updates regarding the issue.

Notes

Getting thread dumps shows that Bamboo is busy getting data from database due to OAuth:

2015-11-11 13:16:07,694 INFO [http-bio-8085-exec-10] [AccessLogFilter] 10.64.38.25 POST http://URL:8085/plugins/servlet/oauth/request-token 173165kb
…
"http-bio-8085-exec-10" #200 daemon prio=5 tid=0x00007f3044003800 nid=0x103b waiting on condition [0x00007f3054a8e000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x00000000d415a3b8> (a com.google.common.util.concurrent.AbstractFuture$Sync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
	at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:317)
	at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
	at com.google.common.util.concurrent.Uninterruptibles.getUninterruptibly(Uninterruptibles.java:132)
	at com.google.common.cache.LocalCache$LoadingValueReference.waitForValue(LocalCache.java:3586)
	at com.google.common.cache.LocalCache$Segment.waitForLoadingValue(LocalCache.java:2333)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2222)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969)
	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829)
	at com.google.common.cache.LocalCache$LocalManualCache.getUnchecked(LocalCache.java:4834)
	at com.atlassian.bamboo.bandana.BambooCachingBandanaPersister.retrieve(BambooCachingBandanaPersister.java:41)
	at com.atlassian.bamboo.bandana.DefaultBandanaPersister.retrieve(DefaultBandanaPersister.java:30)
	at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32)
	at com.atlassian.bamboo.bandana.BambooBandanaManager.getValue(BambooBandanaManager.java:36)
	at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.atlassian.plugin.osgi.hostcomponents.impl.DefaultComponentRegistrar$ContextClassLoaderSettingInvocationHandler.invoke(DefaultComponentRegistrar.java:134)
	at com.sun.proxy.$Proxy40.getValue(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(HostComponentFactoryBean.java:154)
	at com.sun.proxy.$Proxy40.getValue(Unknown Source)
	at com.atlassian.sal.bamboo.pluginsettings.BambooPluginSettings.get(BambooPluginSettings.java:67)
	at com.atlassian.oauth.shared.sal.HashingLongPropertyKeysPluginSettings.migrateKey(HashingLongPropertyKeysPluginSettings.java:56)
	at com.atlassian.oauth.shared.sal.HashingLongPropertyKeysPluginSettings.hashKeyIfTooLong(HashingLongPropertyKeysPluginSettings.java:40)
	at com.atlassian.oauth.shared.sal.HashingLongPropertyKeysPluginSettings.get(HashingLongPropertyKeysPluginSettings.java:29)
	at com.atlassian.oauth.shared.sal.PrefixingPluginSettings.get(PrefixingPluginSettings.java:20)
	at com.atlassian.oauth.serviceprovider.sal.PluginSettingsServiceProviderTokenStore$Settings.getTokenKeySet(PluginSettingsServiceProviderTokenStore.java:450)
	at com.atlassian.oauth.serviceprovider.sal.PluginSettingsServiceProviderTokenStore$Settings.getConsumerTokens(PluginSettingsServiceProviderTokenStore.java:430)
	at com.atlassian.oauth.serviceprovider.sal.PluginSettingsServiceProviderTokenStore$Settings.addConsumerToken(PluginSettingsServiceProviderTokenStore.java:440)
	at com.atlassian.oauth.serviceprovider.sal.PluginSettingsServiceProviderTokenStore$Settings.put(PluginSettingsServiceProviderTokenStore.java:356)
	at com.atlassian.oauth.serviceprovider.sal.PluginSettingsServiceProviderTokenStore.put(PluginSettingsServiceProviderTokenStore.java:166)

2015-11-11 13:00:52,362 INFO [http-bio-8085-exec-11] [AccessLogFilter] 10.64.38.25 POST http://URL:8085/plugins/servlet/oauth/request-token 194800kb
…
"http-bio-8085-exec-11" #215 daemon prio=5 tid=0x00007f309041a000 nid=0x106a runnable [0x00007f2fec6f0000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:170)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:143)
	at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:112)
	at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:71)
	at org.postgresql.core.PGStream.ReceiveChar(PGStream.java:282)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1741)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:405)
	- locked <0x00000000d409d2d0> (a org.postgresql.core.v3.QueryExecutorImpl)
	at org.postgresql.jdbc2.AbstractJdbc2Statement.executeBatch(AbstractJdbc2Statement.java:2892)
	at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeBatch(NewProxyPreparedStatement.java:1723)
	at org.hibernate.jdbc.BatchingBatcher.doExecuteBatch(BatchingBatcher.java:70)
	at org.hibernate.jdbc.AbstractBatcher.executeBatch(AbstractBatcher.java:268)
	at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:268)
	at org.hibernate.engine.ActionQueue.executeActions(ActionQueue.java:185)
	at org.hibernate.event.def.AbstractFlushingEventListener.performExecutions(AbstractFlushingEventListener.java:321)
	at org.hibernate.event.def.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:64)
	at org.hibernate.impl.SessionImpl.autoFlushIfRequired(SessionImpl.java:1185)
	at org.hibernate.impl.SessionImpl.list(SessionImpl.java:1709)
	at org.hibernate.impl.CriteriaImpl.list(CriteriaImpl.java:347)
	at org.hibernate.impl.CriteriaImpl.uniqueResult(CriteriaImpl.java:369)
	at com.atlassian.bamboo.bandana.BandanaItemDaoImpl$1.doInHibernate(BandanaItemDaoImpl.java:29)
	at com.atlassian.bamboo.bandana.BandanaItemDaoImpl$1.doInHibernate(BandanaItemDaoImpl.java:24)
	at org.springframework.orm.hibernate3.HibernateTemplate.doExecute(HibernateTemplate.java:411)
	at org.springframework.orm.hibernate3.HibernateTemplate.execute(HibernateTemplate.java:341)
	at com.atlassian.bamboo.bandana.BandanaItemDaoImpl.getBandanaItem(BandanaItemDaoImpl.java:23)
	at sun.reflect.GeneratedMethodAccessor341.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:98)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:262)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
	at com.sun.proxy.$Proxy59.getBandanaItem(Unknown Source)
	at com.atlassian.bamboo.bandana.BandanaPersisterImpl.retrieve(BandanaPersisterImpl.java:29)
	at com.atlassian.bamboo.bandana.BambooCachingBandanaPersister$1.load(BambooCachingBandanaPersister.java:32)
	at com.atlassian.bamboo.bandana.BambooCachingBandanaPersister$1.load(BambooCachingBandanaPersister.java:28)
	at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3568)
	at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2350)
	at com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2313)
	- locked <0x00000000d40f06b0> (a com.google.common.cache.LocalCache$StrongAccessEntry)
	at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2228)
	at com.google.common.cache.LocalCache.get(LocalCache.java:3965)
	at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3969)
	at com.google.common.cache.LocalCache$LocalManualCache.get(LocalCache.java:4829)
	at com.google.common.cache.LocalCache$LocalManualCache.getUnchecked(LocalCache.java:4834)
	at com.atlassian.bamboo.bandana.BambooCachingBandanaPersister.retrieve(BambooCachingBandanaPersister.java:41)
	at com.atlassian.bamboo.bandana.DefaultBandanaPersister.retrieve(DefaultBandanaPersister.java:30)
	at com.atlassian.bandana.DefaultBandanaManager.getValue(DefaultBandanaManager.java:32)
	at com.atlassian.bamboo.bandana.BambooBandanaManager.getValue(BambooBandanaManager.java:36)
	at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.atlassian.plugin.osgi.hostcomponents.impl.DefaultComponentRegistrar$ContextClassLoaderSettingInvocationHandler.invoke(DefaultComponentRegistrar.java:134)
	at com.sun.proxy.$Proxy40.getValue(Unknown Source)
	at sun.reflect.GeneratedMethodAccessor126.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(HostComponentFactoryBean.java:154)
	at com.sun.proxy.$Proxy40.getValue(Unknown Source)
	at com.atlassian.sal.bamboo.pluginsettings.BambooPluginSettings.get(BambooPluginSettings.java:67)
	at com.atlassian.oauth.shared.sal.HashingLongPropertyKeysPluginSettings.get(HashingLongPropertyKeysPluginSettings.java:29)
	at com.atlassian.oauth.shared.sal.PrefixingPluginSettings.get(PrefixingPluginSettings.java:20)
	at com.atlassian.oauth.serviceprovider.sal.PluginSettingsServiceProviderTokenStore$Settings.getTokenKeySet(PluginSettingsServiceProviderTokenStore.java:450)
	at com.atlassian.oauth.serviceprovider.sal.PluginSettingsServiceProviderTokenStore$Settings.getConsumerTokens(PluginSettingsServiceProviderTokenStore.java:430)
	at com.atlassian.oauth.serviceprovider.sal.PluginSettingsServiceProviderTokenStore$Settings.addConsumerToken(PluginSettingsServiceProviderTokenStore.java:440)
	at com.atlassian.oauth.serviceprovider.sal.PluginSettingsServiceProviderTokenStore$Settings.put(PluginSettingsServiceProviderTokenStore.java:356)
	at com.atlassian.oauth.serviceprovider.sal.PluginSettingsServiceProviderTokenStore.put(PluginSettingsServiceProviderTokenStore.java:166)

2015-11-06 13:19:17,080 INFO [http-bio-8085-exec-518] [AccessLogFilter] 10.64.38.25 GET http://URL:8085/plugins/servlet/streams?use-accept-lang=true&local=true&maxResults=15 191429kb
…
"http-bio-8085-exec-518" #20252 daemon prio=5 tid=0x00007f3090155800 nid=0x5103 in Object.wait() [0x00007f2fe3c09000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	at com.mchange.v2.resourcepool.BasicResourcePool.awaitAvailable(BasicResourcePool.java:1315)
	at com.mchange.v2.resourcepool.BasicResourcePool.prelimCheckoutResource(BasicResourcePool.java:557)
	- locked <0x00000000c2dcefb0> (a com.mchange.v2.resourcepool.BasicResourcePool)
	at com.mchange.v2.resourcepool.BasicResourcePool.checkoutResource(BasicResourcePool.java:477)
	at com.mchange.v2.c3p0.impl.C3P0PooledConnectionPool.checkoutPooledConnection(C3P0PooledConnectionPool.java:525)
	at com.mchange.v2.c3p0.impl.AbstractPoolBackedDataSource.getConnection(AbstractPoolBackedDataSource.java:128)
	at org.hibernate.connection.C3P0ConnectionProvider.getConnection(C3P0ConnectionProvider.java:78)
	at org.hibernate.jdbc.ConnectionManager.openConnection(ConnectionManager.java:446)
	at org.hibernate.jdbc.ConnectionManager.getConnection(ConnectionManager.java:167)
	at org.hibernate.jdbc.BorrowedConnectionProxy.invoke(BorrowedConnectionProxy.java:74)
	at com.sun.proxy.$Proxy358.setReadOnly(Unknown Source)
	at org.springframework.jdbc.datasource.DataSourceUtils.prepareConnectionForTransaction(DataSourceUtils.java:155)
	at org.springframework.orm.hibernate3.HibernateTransactionManager.doBegin(HibernateTransactionManager.java:513)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
	at sun.reflect.GeneratedMethodAccessor127.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:201)
	at com.sun.proxy.$Proxy26.getTransaction(Unknown Source)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:420)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:257)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:95)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
	at com.sun.proxy.$Proxy28.readAclById(Unknown Source)
	at org.acegisecurity.vote.AclEntryVoter.vote(AclEntryVoter.java:210)
	at com.atlassian.bamboo.security.acegi.vote.RestrictedAdminAwareVoter.vote(RestrictedAdminAwareVoter.java:25)
	at org.acegisecurity.vote.UnanimousBased.decide(UnanimousBased.java:63)
	at com.atlassian.bamboo.security.acegi.vote.UnanimousMultiAccessDecisionManager.decide(UnanimousMultiAccessDecisionManager.java:37)
	at org.acegisecurity.intercept.AbstractSecurityInterceptor.beforeInvocation(AbstractSecurityInterceptor.java:292)
	at org.acegisecurity.intercept.web.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:104)
	at org.acegisecurity.intercept.web.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:72)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at com.atlassian.bamboo.filter.SeraphLoginFilter$1.run(SeraphLoginFilter.java:69)
	at com.atlassian.bamboo.filter.SeraphLoginFilter.doFilter(SeraphLoginFilter.java:79)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.ui.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:110)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at org.acegisecurity.util.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:275)
	at org.acegisecurity.util.FilterChainProxy.doFilter(FilterChainProxy.java:149)
	at org.acegisecurity.util.FilterToBeanProxy.doFilter(FilterToBeanProxy.java:98)
	at com.atlassian.bamboo.filter.BambooAcegiProxyFilter.doFilter(BambooAcegiProxyFilter.java:26)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.bamboo.filter.LicenseFilter.doFilter(LicenseFilter.java:76)
	at com.atlassian.core.filters.AbstractHttpFilter.doFilter(AbstractHttpFilter.java:31)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.johnson.filters.AbstractJohnsonFilter.doFilter(AbstractJohnsonFilter.java:71)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.seraph.filter.SecurityFilter.doFilter(SecurityFilter.java:240)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.seraph.filter.BaseLoginFilter.doFilter(BaseLoginFilter.java:148)
	at com.atlassian.seraph.filter.BambooLoginFilter.doFilter(BambooLoginFilter.java:34)
	at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
	at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
	at com.atlassian.plugin.servlet.filter.IteratingFilterChain.doFilter(IteratingFilterChain.java:46)
	at com.atlassian.plugin.servlet.filter.DelegatingPluginFilter$1.doFilter(DelegatingPluginFilter.java:70)
	at com.atlassian.oauth.serviceprovider.internal.servlet.OAuthFilter.doFilter(OAuthFilter.java:61)

Workaround

Disable Bamboo Gadget in JIRA server:

Navigate to Bamboo Administration >> Add-ons >> Manage add-ons >> System add-ons
Disable “Atlassian Bamboo gadgets plugin”
Restart JIRA server because JIRA server cache Bamboo gadget connection details
Bamboo gadget in JIRA server should be disabled now. The gadget will have the following message:

This gadget cannot be displayed on your dashboard. This could be due to a licensing problem or an application error. If you need this gadget, contact your administrator for assistance. Otherwise, you can remove it from your dashboard.
Restart Bamboo to clear the current connections

Attachments

Issue Links

has action: BDEV-11015 Loading...

is related to: OAUTH-321 Loading...

mentioned in: Page Loading...

Performance issue caused by unauthenticated Bamboo gadgets on JIRA dashboard