SimpleScmRequestPoller doesn't correctly handle exceptions

Issue Summary

The SimpleScmRequestPoller runs on a thread named "scm-request-poller". Its job is to repeatedly poll registered futures for SCM requests to see whether they have finished. This is useful because some futures only perform necessary cleanup / notification operations when the future result is retrieved.

The code needs to be robust to exceptions thrown while cleaning up registered future. In the case of an unhandled exception the pollRegisteredFutures() method needs to be resubmitted to the executor service.

Due to a bug, the above mentioned re-submit can only happen once. If the second thread encounters an unhandled exception, a third cleanup task will not be started. This will result in a the growth of command futures, growth in heap memory usage, and eventually an out of memory condition.

Diagnosing this problem can be done via a thread dump. The "scm-request-thread" thread stack should have a frame SimpleScmRequestPoller.pollRegisteredFutures(). For example:

"scm-request-poller:thread-1"
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        at com.atlassian.stash.internal.scm.git.protocol.SimpleScmRequestPoller.pollRegisteredFutures(SimpleScmRequestPoller.java:116)
        - locked <0x0000000787a3d948> (a java.lang.Object)
        at com.atlassian.stash.internal.scm.git.protocol.SimpleScmRequestPoller$$Lambda$812/1091507891.run(Unknown Source)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

a system suffering from this bug will not have a frame with SimpleScmRequestPoller.pollRegisteredFutures() on the "scm-request-thread" thread stack:

"scm-request-poller:thread-2"
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x00000007878319b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
        at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
        at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Steps to Reproduce

<Find a way to have unhandled exceptions thrown in SimpleScmRequestPoller.pollRegisteredFutures()>

Expected Results

SimpleScmRequestPoller continues to cleanup/free command futures after the second unhandled exception.

Actual Results

SimpleScmRequestPoller drank too much booze and stops cleaning up command futures.

Workaround

Currently there is no known workaround for this behavior. A workaround will be added here when available