Automation rules might skip some events if other automation rules are taking more than 5 min to complete

Issue Summary

Automation rules might skip some events if other automation rules are taking more than 5 min to complete.

Environment

Impact any Automation For Jira (A4J) version on Server/Data Center.

Steps to Replicate

1. Create several automation rules to be triggered on the same event (for example "Field Value Changed")
2. Make sure that 1 one these automation rule takes > 5 min to complete (by having it edit a very high number of issues for example)
3. Trigger all the rules together by updating an issue
4. Check the audit logs of all the rules to see if they were all triggered

Expected Results

All the rules should be triggered and process the event.

Actual Results

• Only a few rules are triggered (some rules missed the event)
• Checking the Jira application logs, the exception below is thrown:

  2023-11-07 09:59:18,905+0100 automation-rule-executor:thread-3 ERROR some_user     [c.c.j.p.a.service.execution.JiraThreadLocalExecutor] Unexpected error in thread local executor with actorKey 'some_user'.
  java.lang.IllegalStateException: There is no execution wih id=TenantExecution{tenantExecution=TenantExecution{tenantContext=TenantContext{environment=prod, clientKey='com.codebarrel.tenant.global', tenantId=TenantId{id='00000000-0000-0000-0000-000000bbbbbb'}}, executionUuid='c463420f-d3ae-41cb-9f4c-7d0122769999'}, auditItemId='10360381'}
  	at com.codebarrel.automation.api.execution.DefaultExecutionFinishSynchronizer.lambda$gerRuleInfo$2(DefaultExecutionFinishSynchronizer.java:108)
  	at java.util.Optional.orElseThrow(Optional.java:290)
  	at com.codebarrel.automation.api.execution.DefaultExecutionFinishSynchronizer.gerRuleInfo(DefaultExecutionFinishSynchronizer.java:108)
  	at com.codebarrel.automation.api.execution.DefaultExecutionFinishSynchronizer.arriveAndNotifyIfRequired(DefaultExecutionFinishSynchronizer.java:76)
  	at com.codebarrel.automation.api.service.SingleRuleExecutorServiceImpl.execute(SingleRuleExecutorServiceImpl.java:257)
  	at com.codebarrel.jira.plugin.automation.queue.JiraAutomationQueueExecutor.lambda$processClaimedItem$3(JiraAutomationQueueExecutor.java:275)
  	at com.codebarrel.jira.plugin.automation.service.execution.JiraThreadLocalExecutor.lambda$executeAs$0(JiraThreadLocalExecutor.java:37)
  	at com.codebarrel.jira.plugin.automation.service.execution.JiraThreadLocalExecutor.executeAsWithResult(JiraThreadLocalExecutor.java:67)
  	at com.codebarrel.jira.plugin.automation.service.execution.JiraThreadLocalExecutor.executeAsWithResult(JiraThreadLocalExecutor.java:53)
  	at com.codebarrel.jira.plugin.automation.service.execution.JiraThreadLocalExecutor.executeAs(JiraThreadLocalExecutor.java:36)
  	at com.codebarrel.jira.plugin.automation.queue.JiraAutomationQueueExecutor.lambda$processClaimedItem$4(JiraAutomationQueueExecutor.java:275)
  

Explanation behind this bug

This bug is a combination of 2 factors:

• the fact that events to processed by automation rules are stored in the cache and cleared if they haven't been accessed by a rule for more than 5min
• the fact that one of the rule that is supposed to process this event is taking more than 5 min to complete

When multiple rules need to process the same event, Automation For Jira loops over each rule 1 by 1 to process this event. If one of the rule that is processing the event takes more than 5 min to complete, the following happens:

• the event gets cleared from the cache
• the exception above is thrown in the Jira logs
• Automation For Jira exits the loop even though there were still some rules left to process the event
• ultimately, some rules miss the event and don't get triggered

Workaround

There is unfortunately no ideal/easy workaround for this bug, other than:

• identifying which rule is taking more than 5 min to process the event that was missed by other rules
• either disabling this rule, or optimizing it so that it takes less than 5 min to complete