Development Integration Plugin might create large number of ListenableFutureAdapter threads

Problem Summary

JIRA Software Development Integration Plugin might create large number of ListenableFutureAdapter threads (10k+) and that cause JIRA to stall and crash.

Environment

• Instances with a large amount of issues (1 million+) and linked to (D)VCS accounts.

Steps to Reproduce

1. Create instances with a large amount of issues
2. Link to DVCS accounts with valid content (Bitbucket server or other)
3. Trigger web-hook update-summary to update many issues (5k+) Dev-status

Expected Behavior

JIRA operates normally

Actual Behavior

If large number of issues are requested to update Dev-status, there will be large number ListenableFutureAdapter threads and whole JIRA instance stalls and crashes.

Other symptoms:

• High CPU utilizations
• High disk activity due to active Lucene reindexing.

Notes

• Thread dump collected during that time will show large number of ListenableFutureAdapter threads with following stack:

  "ListenableFutureAdapter-thread-104090" #334429 daemon prio=5 os_prio=0 tid=0x00007fad7867c000 nid=0x6d9a waiting on condition [0x00007fa9db473000]
     java.lang.Thread.State: WAITING (parking)
          at sun.misc.Unsafe.park(Native Method)
          - parking to wait for  <0x00000005cf1b7fb0> (a java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
          at _hostname_19(LinkedBlockingQueue.java:350)
          at com.atlassian.jira.index.QueueingIndex.perform(QueueingIndex.java:74)
          at com.atlassian.jira.issue.index.DefaultIssueIndexer$2.perform(DefaultIssueIndexer.java:198)
  ...
          at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssues(DefaultIndexManager.java:426)
          at com.atlassian.jira.issue.index.DefaultIndexManager.reIndexIssueObjects(DefaultIndexManager.java:455)
  ...
          at com.atlassian.jira.plugin.devstatus.provider.source.cache.DevSummaryStore.reIndex(DevSummaryStore.java:117)
          at com.atlassian.jira.plugin.devstatus.provider.source.cache.CachingProviderDecorator$StoreValueInCache.apply(CachingProviderDecorator.java:321)
          at com.atlassian.jira.plugin.devstatus.provider.source.cache.CachingProviderDecorator$StoreValueInCache.apply(CachingProviderDecorator.java:208)
          at com.google.common.util.concurrent.Futures$4.apply(Futures.java:503)
  ...
          at com.google.common.util.concurrent.JdkFutureAdapters$ListenableFutureAdapter$1.run(JdkFutureAdapters.java:152)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
          at _hostname_2(Thread.java:745)
  

• REST API webhooks /rest/dev-status/1.0/webhooks/update-summary

Workaround

• Reduce number of webhooks call
• Disable Development Integration Plugin (com.atlassian.jira.plugin.devstatus)