Issue Summary

Before the fix for BSERV-20390 was implemented, NutclusterBucketedExecutor dropped a bucket's claimed tasks when the final attempt failed or processing threw an Error, and did not reschedule. As part of the fix for BSERV-20390 we replaced the "drop" logic with a Result flag that starts as FAILURE and is only set to SUCCESS after a successful run. When the last attempt fails, the flag is still FAILURE, so FinishProcessingBucketProcessor returns the claimed tasks to the head of the bucket and the finally block reschedules processing one second later. The bootstrapper's attempt counter is already at max and never increments, so the same tasks are re-claimed and fail again, forever. The Error path loops the same way despite logging "Processing will not be reattempted".

Affects
10.2.3+, 10.3.x, 10.4.x, 10.5.0

Steps to Reproduce

1. Build a BucketedExecutor with maxAttempts = N whose processor always throws
2. Submit one task
3. "Attempt N of N at processing bucket ... failed" is logged every second indefinitely and the bucket is never removed from the IMap

Expected Results

Once a bucket has failed maxAttempts times, its claimed tasks are discarded and processing is not rescheduled.

Actual Results

The failed tasks are returned to the bucket and processing is rescheduled every second indefinitely, logging "Attempt N of N at processing bucket ... failed" at ERROR on each pass until the node is restarted.

Workaround

Currently there is no known workaround for this behavior. A workaround will be added here when available