Issue Summary

PullRequestCommitsReponseObserver leaves it's future incomplete when a RuntimeException is encountered during `onEnd`. The request thread blocks forever waiting for observer.asResult()

This results in the mesh node becoming unresponsive until it is restarded.

This issue was encountered under heavy CI load, where many requests pile up and keep a thread blocked, which eventually exhausts the thread pool. The Node cant recover when load drops.

Steps to Reproduce

  1. `DefaultPullRequestService.streamCommits` wraps the caller's callback in a `BatchingCommitCallback`. The wrapper overrides onCommits to call `PluggableCommitEnricher.enrichAll`.
  2. `BatchingCommitCallback.onEnd` calls `flush(true)`. This sends the last partial batch to enrichAll, which calls HibernateIndexedCommitDao.getProperties.
  3. When the connection pool is exhausted, `getProperties` throws a `RuntimeException`.
  4. `PullRequestCommitsResponseObserver.onCompleted` where it only catches `IOException`.
  5. The RuntimeException leaves onCompleted, and the future is never completed. The gRPC callback executor logs the exception and drops it.
  6. `GrpcPullRequestClient.getCommits` calls `observer.asResult()`. This calls future.get() with no timeout. The gRPC deadline does not help, because the call is already closed.

Expected Results

The RuntimeException is handled exceptionally.

Actual Results

The mesh node eventually becomes unresponsive

Workaround

Currently there is no known workaround for this behavior. A workaround will be added here when available