Issue Summary

One of the Automation service limits is on the count of associated items for a single rule execution, which is capped at 5,000. If this limit is exceeded, the expected behaviour is that:

1. The offending rule does not complete and is marked as Throttled
2. The offending rule is subsequently disabled

Here is what it should __ look like:

However, sometimes there are 2 parts of this that don't behave correctly:

1. Instead of saying THROTTLED the status of the audit is SUCCESS
2. The "Action details:" part of the audit log is completely missing so that the user doesn't understand why the rule was disabled (after what appears to be a successful rule execution)

Steps to Reproduce

This cannot be reliably reproduced since it's intermittent. But in general if you hit the AssItems limit, this should always audit correctly, but perhaps it doesn't always when running locally.

Importantly, here is some useful information to help diagnose the problem and fix. Support case: PCS-264082. 

Logs when the audit log didn't update correctly:

• Splunk logs showing the execution and throttling, but not adding the audit log (search by trace id)
  • Splunk logs for the request id that should __ show that the audit log should be updated (message & status), but it just doesn't.

Expected Results

Explained in more detail in the summary, but TL;DR  Status should be THROTTLED, not SUCCESS, and there should be a long action details summary of what went wrong in the audit log (i.e. user hit the associated items service limit).

Actual Results

The status is SUCCESS, and no mention of hitting the associated items service limit in the audit log.

Workaround

Knowledge of the existence of this bug is the workaround because it's a bug in the audit logging of throttled rules. If the rule is disabled shortly after a successful rule that has > 5000 associated items, then you know that it was throttled; we just haven't made that clear in the UI.

Users in this situation need to amend their rules so that they don't breach the 5000 associated item limit.

Suggested Fix

See Bug diagnosis comment.