Bamboo should exponentially back-off from polling repositories if it encounters repeated failures

Problem Definition

Currently Bamboo will repeatedly poll on whatever interval is specified even when faced with constant and repeated failures.

Constant and repeated failures are common as an instance gets larger across an organization and legacy plans + repositories get left without being removed.

Long timeouts for repositories that are no longer accessible and constantly timeout can tie up important Bamboo threads pools, as well as repeated authentication failures when authentication details are stale or changed continue to add additional system overhead; constantly forking and execing, excessive logging (which has an IO cost too) as we log each error, opening sockets and connections to external servers unnecessarily.

Suggested Solution

Implement failure caching with exponential back-off so that repositories which encounter frequent and repeated failures will not be polled as frequently (up to a configurable maximum).

Re-saving the repository configuration should reset the back-off.

It would be great if the repository itself showed some kind of warning to say that it's been in a repeated failure state (show last error) and back-off mechanism has been applied and a clear application log message is written at WARNING level.