Intermittent resource contention on Automation for Jira cache refresh causing delays on issue update/create

XMLWordPrintable

    • Severity 3 - Minor

      Problem

      Events related to the creation or update of issues that set off A4J rules might sometimes need to pause while the enabledEvents cache in Jira is being refreshed.

      As rules are updated or created throughout the day, this invalidates the existing cache and prompts a refresh on all nodes. The length of time this refresh takes can vary, depending on the number of A4J rules in the instance. Other processes attempting to access the cache will wait for the refresh to complete.

      Review the quick 2-part video for a visualization of how the problem present itself on thread dumps:

      brief overview of A4J cache refresh impact part1.mov

      brief overview of A4J cache refresh impact part2.mov

      Additionally, it would be nice to have this added to the Guardrails on the number of automation rules if there's a point where it becomes an issue.

      Environment

      This was seen on JSW 9.6.1, 9.9.1 and 9.11.2. The suspicion is that it's more visible on instances with:

      • Several thousand A4J rules (longer refresh time fetching from database).
      • High number of rules being created and/or updated.

      Steps to Reproduce

      1. With an instance with a lot of event-based rules (10,000+).
        1. It might be reproducible if a delay in injected on enabledEvents cache refresh.
      2. Update or create a new rule with a non-manual trigger, i.e. issue created or issue updated event.
      3. Immediately send a request to update or create an issue.

      Expected Results

      Jira responds in a timely manner giving back the browser control to the user.

      Actual Results

      See videos from problem description as reference.

      User session will wait for the enabledEvents cache to be refreshed as it's trying to determine which rules to run. Threads are waiting on getRuleConfigBeans until the refresh completes:

      Partial stack trace while waiting:

         java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@11.0.21/Native Method)
        - parking to wait for  <0x00007f59b368f938> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(java.base@11.0.21/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.21/AbstractQueuedSynchronizer.java:885)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(java.base@11.0.21/AbstractQueuedSynchronizer.java:1009)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@11.0.21/AbstractQueuedSynchronizer.java:1324)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(java.base@11.0.21/ReentrantReadWriteLock.java:738)
        at net.sf.ehcache.concurrent.ReadWriteLockSync.lock(ReadWriteLockSync.java:50)
        at net.sf.ehcache.constructs.blocking.BlockingCache.acquiredLockForKey(BlockingCache.java:196)
        at net.sf.ehcache.constructs.blocking.BlockingCache.get(BlockingCache.java:158)
        at com.atlassian.cache.ehcache.LoadingCache.get(LoadingCache.java:120)
        at com.atlassian.cache.ehcache.DelegatingCache.get(DelegatingCache.java:107)
        at com.atlassian.cache.impl.metrics.InstrumentedCache.get(InstrumentedCache.java:72)
        at com.atlassian.jira.cache.stats.CacheWithStats.get(CacheWithStats.java:46)
        at com.codebarrel.jira.plugin.automation.event.EventRulesCacheImpl.getRuleConfigBeans(EventRulesCacheImpl.java:129)
        at com.codebarrel.jira.plugin.automation.event.EventRulesCacheImpl.getRulesForEvent(EventRulesCacheImpl.java:79)
        at com.codebarrel.jira.plugin.automation.event.EventRulesCacheImpl.getAsyncRulesForEvent(EventRulesCacheImpl.java:102)
        at com.codebarrel.jira.plugin.automation.event.DefaultEventRuleRegistry.executeEventRules(DefaultEventRuleRegistry.java:481)

      Workaround

      Option 1

      While the current behavior cannot be worked around, reducing the number of rules either by combining or removing them or limiting rule updates and creation to non-busy hours would reduce the business impact of this scaling problem.

      Option 2

      Since the contention happens when a rule is created or updated, you can use A4J's functionality of allowing/blocking project admins from editing or creating rules. This is documented on this page:

      During days with busier traffic, it's possible to temporarily remove from project admins the ability to manage rules. Once the busy hours are over the permission could be restored.
      Another possibility using the same functionality is to create specific groups to restrict the rule managing ability to a smaller group within your project admins.

      Notes

      I'm adding a few more stack traces and code to increase discoverability of the problem.

      Thread dumps indicated the operations waiting on the enabledCache accessed via getRuleConfigBeans:

      com.codebarrel.jira.plugin.automation.event.EventRulesCacheImpl.getRuleConfigBeans(EventRulesCacheImpl.java:129)
com.codebarrel.jira.plugin.automation.event.EventRulesCacheImpl.getAsyncRules(EventRulesCacheImpl.java:120)

      The thread holding the cache was seen fetching data from database, swapping between these two methods (it has to iterate through all rules):

      Code on line 526

      com.codebarrel.jira.plugin.automation.store.JiraAutomationConfigStore.lambda$groupToRoleConfig$26(JiraAutomationConfigStore.java:526)

                  final List<Long> ruleLabels = dbConnection.newSqlQuery()
                    .select(ruleToLabel.labelId)
                    .from(ruleToLabel)
                    .where(ruleToLabel.ruleId.eq(config.getId()))
                    .fetch();

      Code on line 533

      com.codebarrel.jira.plugin.automation.store.JiraAutomationConfigStore.lambda$groupToRoleConfig$26(JiraAutomationConfigStore.java:533)

                  final List<RuleConfigProjectAssociation> projects = dbConnection.newSqlQuery()
                    .select(ruleConfigProjectAssociation)
                    .from(ruleConfigProjectAssociation)
                    .where(ruleConfigProjectAssociation.ruleId.eq(config.getId()))
                    .fetch();

      The method that leads to them is getRulesConfig:

          private List<RuleConfig> getRuleConfigs(DbConnection dbConnection, BooleanExpression searchExpression) {
        final Map<Long, Group> transform = dbConnection.newSqlQuery()
                .select(ruleConfig, ruleConfigComponent)
                .from(ruleConfig)
                .leftJoin(ruleConfigComponent).on(ruleConfig.id.eq(ruleConfigComponent.ruleId))
                .where(searchExpression)
                .orderBy(ruleConfig.name.asc())
                .orderBy(ruleConfigComponent.sequence.asc())
                .transform(groupBy(ruleConfig.id).as(ruleConfig, list(ruleConfigComponent)));

        return transform.values().stream()
                .map(groupToRoleConfig(dbConnection))
                .filter(Objects::nonNull)
                .collect(Collectors.toList());
    }

      These are the method that clear the cache from DefaultEventRuleRegistry.java:

          public void registerRule(RuleConfigBean rule) {
        if (LOG.isDebugEnabled()) {
            LOG.debug(String.format("Registering event rule with id %d. Clearing cache.", rule.getId()));
        }
        eventRulesCache.clear();
        thirdPartyEventRuleCache.clear();
    }
...
    public void unregisterRule(long ruleId) {
        if (LOG.isDebugEnabled()) {
            LOG.debug(String.format("Unregistering event rule with id %d. Clearing cache.", ruleId));
        }
        eventRulesCache.clear();
        thirdPartyEventRuleCache.clear();
    }
...
    public void unregisterAllRules() {
        LOG.debug("Unregistering all rules.");
        eventRulesCache.clear();
        thirdPartyEventRuleCache.clear();
        shutdownSerializerPool();
    }

      Below is the stack trace with getRuleConfigs. Started from an issue comment that depended on the enabledCache to find the events to trigger. Once it was seen as invalidated, it triggered the refresh.

              at java.net.SocketInputStream.socketRead0(java.base@11.0.21/Native Method)
        at java.net.SocketInputStream.socketRead(java.base@11.0.21/SocketInputStream.java:115)
        at java.net.SocketInputStream.read(java.base@11.0.21/SocketInputStream.java:168)
        at java.net.SocketInputStream.read(java.base@11.0.21/SocketInputStream.java:140)
        at oracle.net.ns.Packet.receive(Packet.java:316)
        at oracle.net.ns.DataPacket.receive(DataPacket.java:128)
        at oracle.net.ns.NetInputStream.getNextPacket(NetInputStream.java:315)
        at oracle.net.ns.NetInputStream.read(NetInputStream.java:257)
        at oracle.jdbc.driver.T4CSocketInputStreamWrapper.read(T4CSocketInputStreamWrapper.java:113)
        at oracle.jdbc.driver.T4CMAREngineStream.getNBytes(T4CMAREngineStream.java:668)
        at oracle.jdbc.driver.T4CMAREngineStream.unmarshalNBytes(T4CMAREngineStream.java:640)
        at oracle.jdbc.driver.DynamicByteArray.unmarshalBuffer(DynamicByteArray.java:295)
        at oracle.jdbc.driver.DynamicByteArray.unmarshalCLR(DynamicByteArray.java:204)
        at oracle.jdbc.driver.T4CClobAccessor.unmarshalPrefetchData(T4CClobAccessor.java:239)
        at oracle.jdbc.driver.T4CClobAccessor.unmarshalBytes(T4CClobAccessor.java:204)
        at oracle.jdbc.driver.T4CClobAccessor.unmarshalOneRow(T4CClobAccessor.java:174)
        at oracle.jdbc.driver.T4CTTIrxd.unmarshal(T4CTTIrxd.java:1594)
        at oracle.jdbc.driver.T4CTTIrxd.unmarshal(T4CTTIrxd.java:1313)
        at oracle.jdbc.driver.T4C8Oall.readRXD(T4C8Oall.java:888)
        at oracle.jdbc.driver.T4CTTIfun.receive(T4CTTIfun.java:468)
        at oracle.jdbc.driver.T4CTTIfun.doRPC(T4CTTIfun.java:269)
        at oracle.jdbc.driver.T4C8Oall.doOALL(T4C8Oall.java:655)
        at oracle.jdbc.driver.T4CPreparedStatement.doOall8(T4CPreparedStatement.java:270)
        at oracle.jdbc.driver.T4CPreparedStatement.fetch(T4CPreparedStatement.java:1079)
        at oracle.jdbc.driver.OracleStatement.fetchMoreRows(OracleStatement.java:3456)
        at oracle.jdbc.driver.InsensitiveScrollableResultSet.fetchMoreRows(InsensitiveScrollableResultSet.java:742)
        at oracle.jdbc.driver.InsensitiveScrollableResultSet.absoluteInternal(InsensitiveScrollableResultSet.java:698)
        at oracle.jdbc.driver.InsensitiveScrollableResultSet.next(InsensitiveScrollableResultSet.java:412)
        - locked <0x00007f58bf2c1fa0> (a oracle.jdbc.driver.T4CConnection)
        at org.apache.commons.dbcp2.DelegatingResultSet.next(DelegatingResultSet.java:1160)
        at org.apache.commons.dbcp2.DelegatingResultSet.next(DelegatingResultSet.java:1160)
        at com.querydsl.sql.SQLResultIterator.hasNext(SQLResultIterator.java:86)
        at com.querydsl.core.group.GroupByMap.transform(GroupByMap.java:56)
        at com.querydsl.core.group.GroupByMap.transform(GroupByMap.java:35)
        at com.querydsl.core.support.FetchableQueryBase.transform(FetchableQueryBase.java:55)
        at com.codebarrel.jira.plugin.automation.store.JiraAutomationConfigStore.getRuleConfigs(JiraAutomationConfigStore.java:589)
        at com.codebarrel.jira.plugin.automation.store.JiraAutomationConfigStore.lambda$getRules$1(JiraAutomationConfigStore.java:122)
        at com.codebarrel.jira.plugin.automation.store.JiraAutomationConfigStore$$Lambda$7347/0x00007f47b620e0a8.runQuery(Unknown Source)
        at com.codebarrel.data.api.jira.JiraDbConnectionManager.lambda$execute$0(JiraDbConnectionManager.java:47)
        at com.codebarrel.data.api.jira.JiraDbConnectionManager$$Lambda$6444/0x00007f47b82b68a8.run(Unknown Source)
        at com.atlassian.jira.database.DatabaseAccessorImpl.executeQuery(DatabaseAccessorImpl.java:77)
        at jdk.internal.reflect.GeneratedMethodAccessor565.invoke(Unknown Source)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.21/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java.base@11.0.21/Method.java:566)
        at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
        at com.sun.proxy.$Proxy512.executeQuery(Unknown Source)
        at jdk.internal.reflect.GeneratedMethodAccessor565.invoke(Unknown Source)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.21/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java.base@11.0.21/Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.doInvoke(ServiceInvoker.java:56)
        at org.eclipse.gemini.blueprint.service.importer.support.internal.aop.ServiceInvoker.invoke(ServiceInvoker.java:60)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invokeUnprivileged(ServiceTCCLInterceptor.java:70)
        at org.eclipse.gemini.blueprint.service.util.internal.aop.ServiceTCCLInterceptor.invoke(ServiceTCCLInterceptor.java:53)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.eclipse.gemini.blueprint.service.importer.support.LocalBundleContextAdvice.invoke(LocalBundleContextAdvice.java:57)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.doProceed(DelegatingIntroductionInterceptor.java:137)
        at org.springframework.aop.support.DelegatingIntroductionInterceptor.invoke(DelegatingIntroductionInterceptor.java:124)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy3643.executeQuery(Unknown Source)
        at com.codebarrel.data.api.jira.JiraDbConnectionManager.execute(JiraDbConnectionManager.java:45)
        at com.codebarrel.jira.plugin.automation.store.JiraAutomationConfigStore.getRules(JiraAutomationConfigStore.java:120)
        at com.codebarrel.jira.plugin.automation.store.JiraAutomationConfigStore.getRules(JiraAutomationConfigStore.java:94)
        at com.codebarrel.jira.plugin.automation.store.CachingAutomationConfigStore.getRules(CachingAutomationConfigStore.java:56)
        at com.codebarrel.automation.api.service.AutomationConfigServiceImpl.getRules(AutomationConfigServiceImpl.java:195)
        at com.codebarrel.jira.plugin.automation.event.EventRulesCacheImpl$1.load(EventRulesCacheImpl.java:55)
        at com.codebarrel.jira.plugin.automation.event.EventRulesCacheImpl$1.load(EventRulesCacheImpl.java:50)
        at com.atlassian.cache.ehcache.wrapper.ValueProcessorAtlassianCacheLoaderDecorator.load(ValueProcessorAtlassianCacheLoaderDecorator.java:26)
        at com.atlassian.cache.ehcache.LoadingCache.getFromLoader(LoadingCache.java:174)
        at com.atlassian.cache.ehcache.LoadingCache$$Lambda$645/0x00007f54a3d1ac40.apply(Unknown Source)
        at com.atlassian.cache.ehcache.SynchronizedLoadingCacheDecorator.synchronizedLoad(SynchronizedLoadingCacheDecorator.java:29)
        at com.atlassian.cache.ehcache.LoadingCache.loadValueAndReleaseLock(LoadingCache.java:142)
        at com.atlassian.cache.ehcache.LoadingCache.get(LoadingCache.java:121)
        at com.atlassian.cache.ehcache.DelegatingCache.get(DelegatingCache.java:107)
        at com.atlassian.cache.impl.metrics.InstrumentedCache.get(InstrumentedCache.java:72)
        at com.atlassian.jira.cache.stats.CacheWithStats.get(CacheWithStats.java:46)
        at com.codebarrel.jira.plugin.automation.event.EventRulesCacheImpl.getRuleConfigBeans(EventRulesCacheImpl.java:129)
        at com.atlassian.cache.ehcache.LoadingCache$$Lambda$645/0x00007f54a3d1ac40.apply(Unknown Source)
        at com.atlassian.cache.ehcache.SynchronizedLoadingCacheDecorator.synchronizedLoad(SynchronizedLoadingCacheDecorator.java:29)
        at com.atlassian.cache.ehcache.LoadingCache.loadValueAndReleaseLock(LoadingCache.java:142)
        at com.atlassian.cache.ehcache.LoadingCache.get(LoadingCache.java:121)
        at com.atlassian.cache.ehcache.DelegatingCache.get(DelegatingCache.java:107)
        at com.atlassian.cache.impl.metrics.InstrumentedCache.get(InstrumentedCache.java:72)
        at com.atlassian.jira.cache.stats.CacheWithStats.get(CacheWithStats.java:46)
        at com.codebarrel.jira.plugin.automation.event.EventRulesCacheImpl.getRuleConfigBeans(EventRulesCacheImpl.java:129)
        at com.codebarrel.jira.plugin.automation.event.EventRulesCacheImpl.getRulesForEvent(EventRulesCacheImpl.java:79)
        at com.codebarrel.jira.plugin.automation.event.EventRulesCacheImpl.getAsyncRulesForEvent(EventRulesCacheImpl.java:102)
        at com.codebarrel.jira.plugin.automation.event.DefaultEventRuleRegistry.executeEventRules(DefaultEventRuleRegistry.java:481)
        at com.codebarrel.jira.plugin.automation.event.DefaultEventRuleRegistry.lambda$handleIssueEventBundle$3(DefaultEventRuleRegistry.java:236)
        at com.codebarrel.jira.plugin.automation.event.DefaultEventRuleRegistry$$Lambda$7332/0x00007f47b6204458.accept(Unknown Source)
        at java.util.ArrayList.forEach(java.base@11.0.21/ArrayList.java:1541)
        at com.codebarrel.jira.plugin.automation.event.DefaultEventRuleRegistry.handleIssueEventBundle(DefaultEventRuleRegistry.java:224)
        at com.codebarrel.jira.plugin.automation.event.JiraIssueEventListenerImpl.handleEvent(JiraIssueEventListenerImpl.java:60)
        at jdk.internal.reflect.GeneratedMethodAccessor3502.invoke(Unknown Source)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.21/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java.base@11.0.21/Method.java:566)
        at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:42)
        at com.atlassian.diagnostics.internal.platform.monitor.event.EventSystemMonitor.invokeMonitored(EventSystemMonitor.java:105)
        at com.atlassian.diagnostics.internal.platform.monitor.event.MonitoredListenerInvoker.invoke(MonitoredListenerInvoker.java:38)
        at com.atlassian.event.internal.ComparableListenerInvoker.invoke(ComparableListenerInvoker.java:48)
        at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.lambda$null$0(AsynchronousAbleEventDispatcher.java:37)
        at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$$Lambda$936/0x00007f54a1dda040.run(Unknown Source)
        at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$$Lambda$343/0x00007f54f73db8c8.execute(Unknown Source)
        at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:85)
        at com.atlassian.diagnostics.internal.platform.monitor.event.MonitoredEventDispatcher.dispatch(MonitoredEventDispatcher.java:36)
        at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:114)
        at com.atlassian.event.internal.LockFreeEventPublisher.publish(LockFreeEventPublisher.java:40)
        at com.atlassian.jira.event.issue.txnaware.TxnAwareEventFactoryImpl.publishEvent(TxnAwareEventFactoryImpl.java:210)
        at com.atlassian.jira.event.issue.txnaware.TxnAwareEventFactoryImpl.lambda$publishOnCommitIssueEventBundle$4(TxnAwareEventFactoryImpl.java:98)
        at com.atlassian.jira.event.issue.txnaware.TxnAwareEventFactoryImpl$$Lambda$7314/0x00007f47b61eb0a8.accept(Unknown Source)
        at com.atlassian.jira.event.issue.txnaware.TxnAwareEventFactoryImpl.lambda$runThisOnCommit$7(TxnAwareEventFactoryImpl.java:189)
        at com.atlassian.jira.event.issue.txnaware.TxnAwareEventFactoryImpl$$Lambda$7302/0x00007f47b61b7840.run(Unknown Source)
        at com.atlassian.ozymandias.SafePluginPointAccess.runnable(SafePluginPointAccess.java:404)
        at com.atlassian.jira.transaction.RunnablesQueueImpl.runIt(RunnablesQueueImpl.java:84)
        at com.atlassian.jira.transaction.RunnablesQueueImpl.runAndClear(RunnablesQueueImpl.java:72)
        at com.atlassian.jira.transaction.TransactionSupportImpl$TransactionImpl.commit(TransactionSupportImpl.java:88)
        at com.atlassian.jira.workflow.OSWorkflowManager.doWorkflowActionInsideTxn(OSWorkflowManager.java:848)
        at com.atlassian.jira.workflow.OSWorkflowManager.doWorkflowAction(OSWorkflowManager.java:799)
        at com.atlassian.jira.bc.issue.DefaultIssueService.transition(DefaultIssueService.java:537)
        at com.atlassian.jira.web.action.issue.CommentAssignIssue.doExecute(CommentAssignIssue.java:173)

      Example stack trace of a thread waiting for the cache to be available:

         java.lang.Thread.State: WAITING (parking)
        at jdk.internal.misc.Unsafe.park(java.base@11.0.21/Native Method)
        - parking to wait for  <0x00007f59b368f938> (a java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(java.base@11.0.21/LockSupport.java:194)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.21/AbstractQueuedSynchronizer.java:885)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(java.base@11.0.21/AbstractQueuedSynchronizer.java:1009)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(java.base@11.0.21/AbstractQueuedSynchronizer.java:1324)
        at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(java.base@11.0.21/ReentrantReadWriteLock.java:738)
        at net.sf.ehcache.concurrent.ReadWriteLockSync.lock(ReadWriteLockSync.java:50)
        at net.sf.ehcache.constructs.blocking.BlockingCache.acquiredLockForKey(BlockingCache.java:196)
        at net.sf.ehcache.constructs.blocking.BlockingCache.get(BlockingCache.java:158)
        at com.atlassian.cache.ehcache.LoadingCache.get(LoadingCache.java:120)
        at com.atlassian.cache.ehcache.DelegatingCache.get(DelegatingCache.java:107)
        at com.atlassian.cache.impl.metrics.InstrumentedCache.get(InstrumentedCache.java:72)
        at com.atlassian.jira.cache.stats.CacheWithStats.get(CacheWithStats.java:46)
        at com.codebarrel.jira.plugin.automation.event.EventRulesCacheImpl.getRuleConfigBeans(EventRulesCacheImpl.java:129)
        at com.codebarrel.jira.plugin.automation.event.EventRulesCacheImpl.getRulesForEvent(EventRulesCacheImpl.java:79)
        at com.codebarrel.jira.plugin.automation.event.EventRulesCacheImpl.getAsyncRulesForEvent(EventRulesCacheImpl.java:102)
        at com.codebarrel.jira.plugin.automation.event.DefaultEventRuleRegistry.executeEventRules(DefaultEventRuleRegistry.java:481)
        at com.codebarrel.jira.plugin.automation.event.DefaultEventRuleRegistry.handleIssueLinkEvent(DefaultEventRuleRegistry.java:342)
        at com.codebarrel.jira.plugin.automation.event.JiraIssueEventListenerImpl.handleIssueLinkCreatedEvent(JiraIssueEventListenerImpl.java:156)
        at jdk.internal.reflect.GeneratedMethodAccessor4451.invoke(Unknown Source)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.21/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java.base@11.0.21/Method.java:566)
        at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:42)
        at com.atlassian.diagnostics.internal.platform.monitor.event.EventSystemMonitor.invokeMonitored(EventSystemMonitor.java:105)
        at com.atlassian.diagnostics.internal.platform.monitor.event.MonitoredListenerInvoker.invoke(MonitoredListenerInvoker.java:38)
        at com.atlassian.event.internal.ComparableListenerInvoker.invoke(ComparableListenerInvoker.java:48)
        at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.lambda$null$0(AsynchronousAbleEventDispatcher.java:37)
        at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$$Lambda$936/0x00007f54a1dda040.run(Unknown Source)
        at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$$Lambda$343/0x00007f54f73db8c8.execute(Unknown Source)
        at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:85)
        at com.atlassian.diagnostics.internal.platform.monitor.event.MonitoredEventDispatcher.dispatch(MonitoredEventDispatcher.java:36)
        at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:114)
        at com.atlassian.event.internal.LockFreeEventPublisher.publish(LockFreeEventPublisher.java:40)
        at com.atlassian.jira.issue.link.DefaultIssueLinkManager.createIssueLink(DefaultIssueLinkManager.java:125)
        at jdk.internal.reflect.GeneratedMethodAccessor4446.invoke(Unknown Source)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.21/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java.base@11.0.21/Method.java:566)
        at com.atlassian.plugin.util.ContextClassLoaderSettingInvocationHandler.invoke(ContextClassLoaderSettingInvocationHandler.java:26)
        at com.sun.proxy.$Proxy394.createIssueLink(Unknown Source)
        at jdk.internal.reflect.GeneratedMethodAccessor4446.invoke(Unknown Source)
        at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(java.base@11.0.21/DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(java.base@11.0.21/Method.java:566)
        at com.atlassian.plugin.osgi.bridge.external.HostComponentFactoryBean$DynamicServiceInvocationHandler.invoke(HostComponentFactoryBean.java:130)
        at com.sun.proxy.$Proxy394.createIssueLink(Unknown Source)
        at com.atlassian.greenhopper.manager.issuelink.EpicLinkManagerImpl.createLink(EpicLinkManagerImpl.java:363)
        at com.atlassian.greenhopper.manager.issuelink.EpicLinkManagerImpl.updateLinksForIssues(EpicLinkManagerImpl.java:289)
        at com.atlassian.greenhopper.manager.issuelink.EpicLinkManagerImpl.associateIssuesWithEpic(EpicLinkManagerImpl.java:242)
        at com.atlassian.greenhopper.customfield.epiclink.EpicLinkCFType.associateIssueWithEpic(EpicLinkCFType.java:857)
        at com.atlassian.greenhopper.customfield.epiclink.EpicLinkCFType.createValue(EpicLinkCFType.java:266)
        at com.atlassian.greenhopper.customfield.epiclink.EpicLinkCFType.createValue(EpicLinkCFType.java:60)
        at com.atlassian.jira.issue.fields.ImmutableCustomField.createValue(ImmutableCustomField.java:701)
        at com.atlassian.jira.issue.fields.ImmutableCustomField.updateValue(ImmutableCustomField.java:414)
        at com.atlassian.jira.issue.fields.ImmutableCustomField.updateValue(ImmutableCustomField.java:400)
        at com.atlassian.jira.issue.managers.DefaultIssueManager.updateFieldValues(DefaultIssueManager.java:715)
        at com.atlassian.jira.issue.managers.DefaultIssueManager.updateIssue(DefaultIssueManager.java:666)
        at com.atlassian.jira.issue.managers.RequestCachingIssueManager.updateIssue(RequestCachingIssueManager.java:222)
        at com.atlassian.jira.bc.issue.DefaultIssueService.update(DefaultIssueService.java:371)
        at com.atlassian.jira.bc.issue.DefaultIssueService.update(DefaultIssueService.java:344)

        1. threads waiting on getRuleConfigBeans.png
          threads waiting on getRuleConfigBeans.png
          5.46 MB
        2. getRulesConfig taking long to complete.png
          getRulesConfig taking long to complete.png
          3.90 MB
        3. brief overview of A4J cache refresh impact part2.mov
          53.86 MB
        4. brief overview of A4J cache refresh impact part1.mov
          46.09 MB

            Assignee:
            Agata Kowal
            Reporter:
            Filipi Lima
            Votes:
            4 Vote for this issue
            Watchers:
            16 Start watching this issue

              Created:
              Updated:
              Resolved: