Summary

Under some rare conditions (for example, unusual activity on the /plugins/servlet/oauth/request-token endpoint), threads may become deadlocked in database queries involving the plugin_setting table. This deadlock ties up connections to the database and increases the likelihood that further threads will also become deadlocked. In extreme cases, threads may no longer be able to obtain connections from the DB connection pool, and the instance may require restarting.

Symptoms

If affected, the number of open connections to the database does not return to 0 even when idle, but increases monotonically. It is possible to monitor the number of open database connections in Stash or Bitbucket Server directly using the following JMX beans:

• Stash 3.8 to 3.11, com.jolbox.bonecp.BoneCP.TotalLeased as described in https://confluence.atlassian.com/display/STASH0311/Enabling+JMX+counters+for+performance+monitoring
• Bitbucket Server 4.0 or higher, com.zaxxer.hikari.Pool (bitbucket).ActiveConnections as described in https://confluence.atlassian.com/bitbucketserver/enabling-jmx-counters-for-performance-monitoring-776640189.html

If affected, a Stash instance may remain up for a while (days or longer) with no adverse effects except for a mildly elevated number of open database connections. But once deadlocked, the risk of further threads becoming deadlocked on the same resource increases. If very many threads become deadlocked, the DB connection pool (with 80 connections by default) may run out and the instance will require restarting.

An analysis of the open transactions on the database (for example, with select * from pg_stat_activity order by xact_start; in PostgreSQL) while Stash is affected by this issue will show a number of (possibly very old) connections stuck in queries on the plugin_setting table, for example:

delete from plugin_setting where id=$1
update plugin_setting set key_name=$1, namespace=$2, key_value=$3 where id=$4

Thread dumps of Stash taken while affected by this issue may also contain many threads blocked with com.atlassian.stash.internal.plugin.CachingPluginSettings classes mentioned in the stack traces.

Workarounds

1. Avoid integrations which make frequent requests on the /plugins/servlet/oauth/request-token endpoint. Atlassian products generally only make requests to this endpoint once every few hours, but third party integrations may make such requests more frequently. You can check your atlassian-stash-access-*.log or atlassian-bitbucket-access-*.log files to find such activity.
2. Avoid plugins that perform PluginSettings API calls in large transactions. Note that the bundled Atlassian OAuth Plugin can perform very many PluginSettings API calls under some conditions, see the related issue https://ecosystem.atlassian.net/browse/OAUTH-321.
3. Stash 3.11.6 contains a mitigation that makes this deadlock exceedingly unlikely to occur. (See BSERV-8522)
4. If affected, forcibly terminating very old transactions stuck on queries on the plugin_setting table at the database level may clear the deadlock without requiring Stash or Bitbucket Server to be restarted.

Solution

Bitbucket Server 4.1.0 contains a solution that prevents the deadlock from occurring.