Service limit breached trigger rules might cause queue issues

XMLWordPrintable

    • Severity 3 - Minor

      Issue Summary

      The Service limit breached trigger is intended to warn admins when the limit is breached so they can take action.

      However, this trigger has throttled rules insert an item in the queue for every throttled rule execution. When the system throttles due to being under pressure, this increases the queue count, creating a bottleneck on the automation-queue-claimer thread.

      This happens even if the rules are disabled.

      Steps to Reproduce

      1. Create a rule that triggers when an issue is created;
        For example, create a rule that creates a task whenever a bug is created and link them.
      2. Create two rules with the Service limit breached trigger;
      3. Lower the max.processing.time.per.day threshold, or saturate the rule by calling it many times (that is, creating many bugs in this example);
      4. Run a loop to create many bugs so we put pressure on the system.
        Here's an example of how to create 5 threads that create in parallel, but it might take considerably more, depending on the system resources.
        for i in $(seq 1 300); do curl -s -o /dev/null -u admin:password -X POST --data '{"fields":{"project":{"key": "KANBAN" },"summary": "Test thread 1 - '$i'","issuetype": {"name": "Bug"}}}' -H "Content-Type: application/json" https://myjira.com/rest/api/2/issue ;done &
        for i in $(seq 1 300); do curl -s -o /dev/null -u admin:password -X POST --data '{"fields":{"project":{"key": "KANBAN" },"summary": "Test thread 2 - '$i'","issuetype": {"name": "Bug"}}}' -H "Content-Type: application/json" https://myjira.com/rest/api/2/issue ;done &
        for i in $(seq 1 300); do curl -s -o /dev/null -u admin:password -X POST --data '{"fields":{"project":{"key": "KANBAN" },"summary": "Test thread 3 - '$i'","issuetype": {"name": "Bug"}}}' -H "Content-Type: application/json" https://myjira.com/rest/api/2/issue ;done &
        for i in $(seq 1 300); do curl -s -o /dev/null -u admin:password -X POST --data '{"fields":{"project":{"key": "KANBAN" },"summary": "Test thread 4 - '$i'","issuetype": {"name": "Bug"}}}' -H "Content-Type: application/json" https://myjira.com/rest/api/2/issue ;done &
        for i in $(seq 1 300); do curl -s -o /dev/null -u admin:password -X POST --data '{"fields":{"project":{"key": "KANBAN" },"summary": "Test thread 5 - '$i'","issuetype": {"name": "Bug"}}}' -H "Content-Type: application/json" https://myjira.com/rest/api/2/issue ;done &
        

      Expected Results

      A4J queue (AO_589059_AUTOMATION_QUEUE) only has queue items for the rules that should run.

      Actual Results

      We can see that throttled executions insert items in the queue for each

      2026-01-22 14:05:54,827+0000 automation-rule-executor:thread-1 DEBUG admin     [c.c.a.api.queue.AutomationQueueImpl] Adding 2 items to the queue (only showing first 3): [QueueItemBean{id=0, auditItemId=null, executionUuid=35497c5a-e06b-4eb6-86d3-b6b6a8d4b59d, ruleId=3, priority=0, created=Thu Jan 22 14:05:54 UTC 2026, claimant='null', claimTime=null, payload='{"tenantContext":{"clientKey":"com.codebarrel.tenant.global","environment":"prod","tenantId":{"id":"00000000-0000-0000-0000-000000bbbbbb"}},"event":{"clientKey":"com.codebarrel.tenant.global","triggerType":"SYSTEM","type":"automation.trigger.servi...'}, QueueItemBean{id=0, auditItemId=null, executionUuid=70f67d50-f712-4510-b405-a558f2d537ca, ruleId=2, priority=0, created=Thu Jan 22 14:05:54 UTC 2026, claimant='null', claimTime=null, payload='{"tenantContext":{"clientKey":"com.codebarrel.tenant.global","environment":"prod","tenantId":{"id":"00000000-0000-0000-0000-000000bbbbbb"}},"event":{"clientKey":"com.codebarrel.tenant.global","triggerType":"SYSTEM","type":"automation.trigger.servi...'}]
      

      Those queue items have priority zero, so they slow down other queue items from being processed and also they should mostly not pass the trigger validation, as they're supposed to be processed only once every hour.
      As that's the case, the rule executors probably should first verify if it makes sense to add these service limit breach executions to the queue in the first place.

      Additionally, the DEBUG logs only show enabled rules having their executions queued, but verifying the automation queue in the database, we can see execution instances of disabled rules for this trigger type.
      These executions don't pass validation when they start, but they use up resources by being added to the queue.

      Workaround

      Locate the rules with these triggers with this query:

      SELECT rd."ID",rc."NAME"
      FROM "AO_589059_RULE_CONFIG" rc
      JOIN "AO_589059_RULE_CFG_COMPONENT" rcc ON rcc."RULE_CONFIG_ID"=rc."ID"
      WHERE rcc."TYPE"='automation.trigger.service.limit'
      

      Navigate to the endpoint /secure/AutomationGlobalAdminAction!default.jspa#/rule/RULE_ID_HERE in a new tab to go to each rule and delete it.

      You can monitor Automation for Jira events by using the metrics described in https://support.atlassian.com/jira/kb/automation-for-jira-monitoring-automation-queue/
      Also, the queue count (automation.queue.count.max_stale_10mins) can be fetched from the atlassian-jira-perf.log, by using the commands from either https://support.atlassian.com/jira/kb/how-to-manually-parse-the-atlassian-jira-perflog-performance-log-from-jira/ or https://support.atlassian.com/jira/kb/how-to-dynamically-parse-jira-performance-log-used-for-gathering-jmx-statistics/

              Assignee:
              Unassigned
              Reporter:
              Allan Gandelman
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated: