Provide an isolated executor for Bitbucket core logic instead of relying on the ScheduledExecutorService

Problem Definition

A shared ScheduledExecutorService is made available for add-ons to use, and add-on developers are encouraged to use it instead of, for example, doing heavy processing directly in event listeners (which are invoked on a limited number of dispatcher threads).

However, there are some core part of the application that are also using the same shared executor.
That means if add-ons invoke particularly slow/heavy operations on the shared executor, it can block the operation that the core part of the application is performing.

So far, this problem has been observed in the following functionalities:

• Time outs when displaying branches:

  c.a.s.i.r.PluginRefMetadataMapProvider Timed out when retrieving ref metadata for com.atlassian.bitbucket.server.bitbucket-jira:branch-list-jira-issues
  c.a.s.i.r.PluginRefMetadataMapProvider Timed out when retrieving ref metadata for com.atlassian.bitbucket.server.bitbucket-branch:ahead-behind-metadata-provider
  c.a.s.i.r.PluginRefMetadataMapProvider Timed out when retrieving ref metadata for com.atlassian.bitbucket.server.bitbucket-branch:latest-commit-metadata
  c.a.s.i.r.PluginRefMetadataMapProvider Timed out when retrieving ref metadata for com.atlassian.bitbucket.server.bitbucket-build:build-status-metadata
  

• HookService, which handles pre-receive and post-receive callbacks from pushes, to cause slow pushes (Handled separately on BSERV-10652)
• Diff transcoding triggering a backlog of transcode.pl processes, which may manifest as slow blame or diff views
• Processing notifications to create emails

Common Misconceptions

Sending email is not handled by the shared ScheduledExecutorService. It's handled by a separate, dedicated executor which, by default, has a single thread. If emails are building up in the queue the number of threads dedicated to sending emails can be increased by adding an entry like the following in bitbucket.properties and restarting Bitbucket Server:

mail.threads=4 

Note that 4 is just an example, not a recommendation.

Suggested Solution

The affected services should be updated to use their own executor. This would ensure that their processing would not be delayed by other parts of the system.

Workaround

Disable/uninstall the add on causing the high and slow usage of the shared executor.