Database deadlock when updating plugin

Problem

In some cases when UPM is trying to update a plugin, a deadlock occurs in the database.

The issue can be reliably reproduced with plugins that are published in the Atlassian Marketplace:

• environment: Confluence 6.7.1, 6.7.2, or 6.8.1, PostgreSQL 9.3.5
• install Scroll PDF Exporter 4.0.18
• install Scroll Versions 3.8.5
• update Scroll Versions to 3.8.7 (the latest version at the time of writing)

Relevant logs:

[p=39237, tx=5396419, vtx=7/1353] LOG:  execute S_4: update BANDANA set BANDANACONTEXT=$1, BANDANAKEY=$2, BANDANAVALUE=$3 where BANDANAID=$4
[p=39237, tx=5396419, vtx=7/1353] DETAIL:  parameters: $1 = '_GLOBAL', $2 = 'plugin.manager.state.Map', $3 = '<map>
	  <entry>
	    <string>com.atlassian.confluence.plugins.confluence-onboarding:onboarding</string>
	    <boolean>false</boolean>
	  </entry>
	  <entry>
	    <string>com.k15t.scroll.scroll-exporter-extensions:scroll.only</string>
	    <boolean>false</boolean>
	  </entry>
	</map>', $4 = '8'
[p=39237, tx=5396419, vtx=7/1353] ERROR:  deadlock detected
[p=39237, tx=5396419, vtx=7/1353] DETAIL:  Process 39237 waits for ShareLock on transaction 5396429; blocked by process 39238.
	Process 39238 waits for ShareLock on transaction 5396419; blocked by process 39237.
	Process 39237: update BANDANA set BANDANACONTEXT=$1, BANDANAKEY=$2, BANDANAVALUE=$3 where BANDANAID=$4
	Process 39238: update BANDANA set BANDANACONTEXT=$1, BANDANAKEY=$2, BANDANAVALUE=$3 where BANDANAID=$4

where:

• 39237 pertains to vendor transaction's update (select, update from PLUGINDATA and BANADA)
• 39238 pertains to atlassian.confluence.plugin.counter

It is not apparent that there is anything wrong with those plugins. It rather seems that something goes wrong when Confluence tries to dispatch the plugin install event. Further investigation shows that the Gadget Plugin (com.atlassian.confluence.plugins.gadgets), which is a system plugin, is somehow involved in the deadlock. Disabling the Gadget Plugin for the duration of the update prevents the problem.

When the deadlock occurs, the log file is flooded with exceptions. Attached is a sample. Also attached, a PostgreSQL log file showing the events leading up to the deadlock (filtered for the two transactions involved).

I'm assuming that the Gadget Plugin causes the issue, because it is implicated both in the exception stacktraces and the PostgreSQL logs.

However, the problem also seems contingent on the specific versions of those user-installed plugins. I don't excactly know why. Interestingly, the problem does not occur :

• if version 4.1.0 of Scroll PDF Exporter is installed.
• If the Scroll Exporter Extensions plugin, which comes bundled with the Scroll PDF Exporter and contains a number of macros, is disabled, the problem also does not occur. The new version of Scroll Versions bundles a new version of the Scroll Runtime for Confluence plugin, which both Scroll Versions and Scroll PDF Exporter depend on.

The Scroll Exporter Extensions plugin that is bundled in Scroll PDF Exporter 4.1.0 also depends on the Scroll Runtime for Confluence (it didn't before), which means that if 4.1.0 is installed, the Scroll Exporter Extensions plugin is temporarily disabled during the update. Does the Gadget Plugin somehow mess with the macro modules of the extension plugin during the update?

Another theory we had was that the optional require-bundle dependency of Scroll PDF Exporter on Scroll Exporter Extensions could trigger the issue, but I was unable to corroborate that theory. Replacing it with a mandatory import-package dependency did not change anything.

Another observation: the actual update succeeds in that the new versions of the plugins are installed afterwards, even though UPM says that there was an error. But I'm not sure if all events, migration/upgrade tasks can complete successfully in such a case.