Uploaded image for project: 'Bitbucket Server'
  1. Bitbucket Server
  2. BSERV-12808

Searching in Bitbucket results in deadlocked threads

    XMLWordPrintable

    Details

      Description

      Issue Summary

      On attempting to search for results in Bitbucket, it's possible that the thread handling the search request can get into a deadlock, permanently increasing the number of concurrently executing threads until the JVM is restarted.

      Steps to Reproduce

      • The exact conditions for when this issue can occur are not currently known. It's currently estimated to be related to the configured Elasticsearch instance being unreachable for an extended period of time following a restart of the Bitbucket instance.

      Expected Results

      One of the following happens:

      • Bitbucket successfully completes the search request
      • Bitbucket gets back an error and logs it/sends it back to the entity making the request
      • Bitbucket times out in waiting for a response and logs it and reports the timeout to the entity making the request

      Actual Results

      The thread becomes stuck in a deadlock. On taking a series of thread dumps, the following type of WAITING threads can be seen:

      "https-jsse-nio-8443-exec-178" #125950 daemon prio=5 os_prio=0 tid=0x00007f51b013d800 nid=0x16f01 waiting on condition [0x00007f517fe0c000]
         java.lang.Thread.State: WAITING (parking)
      	at sun.misc.Unsafe.park(Native Method)
      	- parking to wait for  <0x00000000ad69a1a8> (a java.util.concurrent.CountDownLatch$Sync)
      	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
      	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
      	at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
      	at com.atlassian.bitbucket.internal.search.indexing.util.Observables.consume(Observables.java:75)
      	at com.atlassian.bitbucket.internal.search.indexing.util.Observables.consumeSingle(Observables.java:92)
      	at com.atlassian.bitbucket.internal.search.search.rest.SearchResource.processRequest(SearchResource.java:340)
      	at com.atlassian.bitbucket.internal.search.search.rest.SearchResource.lambda$search$5(SearchResource.java:112)
      	at com.atlassian.bitbucket.internal.search.search.rest.SearchResource$$Lambda$3322/193574928.apply(Unknown Source)
      ...
      

      Workaround

      Restart the Bitbucket instance to clear up the stuck threads, and then confirm that the configured Elasticsearch instance is both reachable and able to be successfully queried.

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              mgoyal2@atlassian.com Manish
              Reporter:
              eslaughter@atlassian.com Evan Slaughter
              Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Backbone Issue Sync