Uploaded image for project: 'Bitbucket Server'
  1. Bitbucket Server
  2. BSERV-12205

SimpleScmRequestPoller doesn't correctly handle exceptions

    XMLWordPrintable

    Details

      Description

      Issue Summary

      The SimpleScmRequestPoller runs on a thread named "scm-request-poller". Its job is to repeatedly poll registered futures for SCM requests to see whether they have finished. This is useful because some futures only perform necessary cleanup / notification operations when the future result is retrieved.

      The code needs to be robust to exceptions thrown while cleaning up registered future. In the case of an unhandled exception the pollRegisteredFutures() method needs to be resubmitted to the executor service.

      Due to a bug, the above mentioned re-submit can only happen once. If the second thread encounters an unhandled exception, a third cleanup task will not be started. This will result in a the growth of command futures, growth in heap memory usage, and eventually an out of memory condition.

      Diagnosing this problem can be done via a thread dump. The "scm-request-thread" thread stack should have a frame SimpleScmRequestPoller.pollRegisteredFutures(). For example:

      "scm-request-poller:thread-1"
         java.lang.Thread.State: WAITING (on object monitor)
              at java.lang.Object.wait(Native Method)
              at com.atlassian.stash.internal.scm.git.protocol.SimpleScmRequestPoller.pollRegisteredFutures(SimpleScmRequestPoller.java:116)
              - locked <0x0000000787a3d948> (a java.lang.Object)
              at com.atlassian.stash.internal.scm.git.protocol.SimpleScmRequestPoller$$Lambda$812/1091507891.run(Unknown Source)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      

      a system suffering from this bug will not have a frame with SimpleScmRequestPoller.pollRegisteredFutures() on the "scm-request-thread" thread stack:

      "scm-request-poller:thread-2"
         java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0x00000007878319b0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
              at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
              at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      

      Steps to Reproduce

      <Find a way to have unhandled exceptions thrown in SimpleScmRequestPoller.pollRegisteredFutures()>

      Expected Results

      SimpleScmRequestPoller continues to cleanup/free command futures after the second unhandled exception.

      Actual Results

      SimpleScmRequestPoller drank too much booze and stops cleaning up command futures.

      Workaround

      Currently there is no known workaround for this behavior. A workaround will be added here when available

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              behumphreys Ben Humphreys
              Reporter:
              behumphreys Ben Humphreys
              Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: