Issue Summary

Outgoing Opsgenie to Zendesk API calls have been periodically hitting Zendesk's Rate Limits and thus failing with the 429 error.  Two of the most common 429 errors are "setStatusToSolved" and "addInternalComment"

This has resulted in thousands of Zendesk tickets left unclosed, and out-of-sync with Opsgenie Alerts status.  

Recent episodes of these 429 errors lasted about 1 to 3 minutes each time and happens almost daily between 1 and 3 times a day.

Steps to Reproduce

1. PRTG creates the Opsgenie alerts that is sent to Zendesk
2. Whenever PRTG closes an alert, it also adds a comment to the alert.  Both these  transactions (setStatusToSolved and addInternalComment) are sent from Opsgenie to Zendesk 

Expected Results

Per Zendesk Rate Limits document, you can use the following response headers to confirm the [customer's Zendesk] account's current rate limit
 
X-Rate-Limit: 700
X-Rate-Limit-Remaining: 699

"If the rate limit is exceeded, the API responds with a 429 Too Many Requests status code. The response also has a Retry-After header that tells you how many seconds to wait before retrying API requests. Ensure your code handles 429 errors and waits the Retry-After interval before retrying requests."

The expectation is for the Opsgenie Zendesk integration "to have a fazed commit queue" to avoid hitting the rate limit. The alert storm is due to outage.  The expectation is that the system has to handle the burst of activity without leaving danglers.

Actual Results

Most of the setStatusToSolved and addInternalComment were "successfully sent", however, periodically some would "failed to send".  The below exceptions is thrown in the Logs (under Settings):

[system] [ZendeskV2-Opsgenie Alerts to Zendesk Tickets (Support)] Failed to send [setStatusToSolved], reason: 429 Too Many Requests: "{"errors":[{"code":"tickets-update","title":"Ticket update API threshold exceeded","status":429}]}"

system] [ZendeskV2-Opsgenie Alerts to Zendesk Tickets (Support)] Failed to send [addInternalComment], reason: 429 Too Many Requests: "{"errors":[{"code":"tickets-update","title":"Ticket update API threshold exceeded","status":429}]}"

[ZendeskV2-Opsgenie Alerts to Zendesk Tickets (Support)] Failed to send [setStatusToSolved], will not retry.
 exception: org.springframework.web.client.HttpClientErrorException$TooManyRequests: 429 Too Many Requests: "{"errors":[{"code":"tickets-update","title":"Ticket update API threshold exceeded","status":429}]}" 

Workaround

Currently there is no known workaround for this behavior, other than subscribing to a Zendesk Suite plan with a higher rate limit (where available) or use additional filters to limit the number of tickets created in Zendesk.