Repository cache isn't utilized upon receiving remote events from Bitbucket

XMLWordPrintable

    • Severity 2 - Major

      Summary

      Repository cache isn't utilized upon receiving remote events from Bitbucket including:

      • Branch changed remote event
      • Mirror synced remote event

      This can put excessive load on the database as each time an event is received the below SELECT needs to be executed on VCS_LOCATION (which can get rather large due to XML data being stored per master repository row):

      select this_.* from VCS_LOCATION this_ 
where this_.PLUGIN_KEY='com.atlassian.bamboo.plugins.stash.atlassian-bamboo-plugin-stash:bbserver'

      In some situations, it can end up with 16 concurrent selects are running on a very large table with a full tablescan.

      Steps to Reproduce

      These are purely faked test steps to reproduce the issue of always going to the database and not a real world example.

      1. Fresh instance with one Bitbucket repository and one plan with a Bitbucket Server Repository triggered trigger.
      2. Lock the VCS_LOCATION table with exclusive lock so that any SELECT on it hang: 
        --- Lock table (PostgreSQL)
@set autocommit off;
LOCK TABLE VCS_LOCATION IN ACCESS EXCLUSIVE MODE;
SELECT count(vcs_location_id) FROM VCS_LOCATION;
      3. Commit to a Bitbucket repository twice so it sends 2x stash-branch-changed-remote-event

      FYI can be unlocked with:

      --- Unlock table
commit transaction;

      Expected Results

      Repositories are pulled from cache rather than going to the database to query VCS_LOCATION

      Actual Results

      AtlassianEvent threads hang in the same spot waiting for the results of SQL query:

      	at com.atlassian.bamboo.repository.RepositoryDefinitionHibernateDao.lambda$findRepositoriesByPluginKey$4(RepositoryDefinitionHibernateDao.java:133)

      Full stack:

      "AtlassianEvent::0-BAM::EVENTS:pool-1-thread-13" #190 daemon prio=1 os_prio=0 tid=0x00007f3f849a0800 nid=0x67bd runnable [0x00007f3f10470000]
   java.lang.Thread.State: RUNNABLE
	at java.net.SocketInputStream.socketRead0(Native Method)
	at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
	at java.net.SocketInputStream.read(SocketInputStream.java:171)
	at java.net.SocketInputStream.read(SocketInputStream.java:141)
	at org.postgresql.core.VisibleBufferedInputStream.readMore(VisibleBufferedInputStream.java:140)
	at org.postgresql.core.VisibleBufferedInputStream.ensureBytes(VisibleBufferedInputStream.java:109)
	at org.postgresql.core.VisibleBufferedInputStream.read(VisibleBufferedInputStream.java:67)
	at org.postgresql.core.PGStream.receiveChar(PGStream.java:282)
	at org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.java:1898)
	at org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java:297)
	- locked <0x00000000d059a200> (a org.postgresql.core.v3.QueryExecutorImpl)
	at org.postgresql.jdbc.PgStatement.executeInternal(PgStatement.java:428)
	at org.postgresql.jdbc.PgStatement.execute(PgStatement.java:354)
	at org.postgresql.jdbc.PgPreparedStatement.executeWithFlags(PgPreparedStatement.java:169)
	at org.postgresql.jdbc.PgPreparedStatement.executeQuery(PgPreparedStatement.java:117)
	at com.mchange.v2.c3p0.impl.NewProxyPreparedStatement.executeQuery(NewProxyPreparedStatement.java:353)
	at org.hibernate.engine.jdbc.internal.ResultSetReturnImpl.extract(ResultSetReturnImpl.java:71)
	at org.hibernate.loader.Loader.getResultSet(Loader.java:2123)
	at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1911)
	at org.hibernate.loader.Loader.executeQueryStatement(Loader.java:1887)
	at org.hibernate.loader.Loader.doQuery(Loader.java:932)
	at org.hibernate.loader.Loader.doQueryAndInitializeNonLazyCollections(Loader.java:349)
	at org.hibernate.loader.Loader.doList(Loader.java:2615)
	at org.hibernate.loader.Loader.doList(Loader.java:2598)
	at org.hibernate.loader.Loader.listIgnoreQueryCache(Loader.java:2430)
	at org.hibernate.loader.Loader.list(Loader.java:2425)
	at org.hibernate.loader.criteria.CriteriaLoader.list(CriteriaLoader.java:109)
	at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1896)
	at org.hibernate.internal.CriteriaImpl.list(CriteriaImpl.java:366)
	at com.atlassian.bamboo.repository.RepositoryDefinitionHibernateDao.lambda$findRepositoriesByPluginKey$4(RepositoryDefinitionHibernateDao.java:133)
	at com.atlassian.bamboo.repository.RepositoryDefinitionHibernateDao$$Lambda$736/1652144711.doInHibernate(Unknown Source)
	at org.springframework.orm.hibernate5.HibernateTemplate.doExecute(HibernateTemplate.java:359)
	at org.springframework.orm.hibernate5.HibernateTemplate.execute(HibernateTemplate.java:313)
	at com.atlassian.bamboo.repository.RepositoryDefinitionHibernateDao.findRepositoriesByPluginKey(RepositoryDefinitionHibernateDao.java:131)
	at com.atlassian.bamboo.repository.RepositoryDefinitionManagerImpl.findRepositoriesByPluginKey(RepositoryDefinitionManagerImpl.java:242)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at com.atlassian.bamboo.security.acegi.intercept.aopalliance.AuthorityOverrideMethodSecurityInterceptor.invoke(AuthorityOverrideMethodSecurityInterceptor.java:24)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:282)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
	at com.sun.proxy.$Proxy139.findRepositoriesByPluginKey(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
	at com.sun.proxy.$Proxy513.findRepositoriesByPluginKey(Unknown Source)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
	at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
	at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:133)
	at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:121)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:207)
	at com.sun.proxy.$Proxy1345.findRepositoriesByPluginKey(Unknown Source)
	at com.atlassian.bamboo.plugins.stash.events.RepositoryFinder.findMatchingRepositories(RepositoryFinder.java:46)
	at com.atlassian.bamboo.plugins.stash.events.BranchChangedEventListener.handleRefsChange(BranchChangedEventListener.java:165)
	at com.atlassian.bamboo.plugins.stash.events.BranchChangedEventListener.onBranchChangedEvent(BranchChangedEventListener.java:155)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:36)
	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:46)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at com.atlassian.bamboo.utils.BambooRunnables$1.run(BambooRunnables.java:47)
	at com.atlassian.bamboo.security.ImpersonationHelper.runWith(ImpersonationHelper.java:26)
	at com.atlassian.bamboo.security.ImpersonationHelper.runWithSystemAuthority(ImpersonationHelper.java:17)
	at com.atlassian.bamboo.security.ImpersonationHelper$1.run(ImpersonationHelper.java:41)
	at java.lang.Thread.run(Thread.java:748)

      Workaround

      Currently there is no known workaround for this behavior. A workaround will be added here when available.

            Assignee:
            Przemek Bruski
            Reporter:
            Jeremy Owen
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

              Created:
              Updated:
              Resolved: