-
Bug
-
Resolution: Timed out
-
Medium
-
None
-
4.0, 4.3
-
Confluence standalone; MySQL 5.0
-
3
-
Severity 2 - Major
-
As originally described in my comments to CONF-21986, Confluence has a bad habit of reporting deadlocks and failing to handle plugin installations gracefully.
This was observed when trying to upgrade the Adaptavist Plugin Licensing Manager plugin to v0.6.1, while running a recently-upgraded Confluence 4.0 in a production environment:
2011-09-23 11:51:39,838 INFO [Spring executor 11] [atlassian.plugin.manager.DefaultPluginManager] notifyPluginDisabled Disabling com.adaptavist.confluence.contentFormattingMacros 2011-09-23 11:51:43,336 INFO [Spring executor 11] [atlassian.plugin.manager.DefaultPluginManager] disableDependentPlugins Found dependent enabled plugins for uninstalled plugin 'com.adaptavist.plm.plugin.plm-plugin': []. Disabling... 2011-09-23 11:51:43,337 INFO [Spring executor 11] [atlassian.plugin.manager.DefaultPluginManager] notifyPluginDisabled Disabling com.adaptavist.plm.plugin.plm-plugin 2011-09-23 11:52:35,488 ERROR [Spring executor 11] [sf.hibernate.util.JDBCExceptionReporter] logExceptions Lock wait timeout exceeded; try restarting transaction -- referer: http://172.16.16.10:8090/confluence/plugins/servlet/upm | url: /confluence/rest/plugins/1.0/upgrades/all | userName: scott 2011-09-23 11:52:35,490 ERROR [Spring executor 11] [sf.hibernate.impl.SessionImpl] execute Could not synchronize database state with session -- referer: http://172.16.16.10:8090/confluence/plugins/servlet/upm | url: /confluence/rest/plugins/1.0/upgrades/all | userName: scott 2011-09-23 11:52:40,333 INFO [pool-6-thread-1] [atlassian.plugin.manager.DefaultPluginManager] notifyPluginDisabled Disabling com.adaptavist.plm.plugin.plm-plugin-installer
We initially discovered what appears to be the same issue when developing one of our own plugins, which may shed some additional light on the problem. It seems that our plugin's Spring initialization is getting invoked while Confluence still holds a lock on some part of the plugin table. Our initialization code sometimes tries to modify other plugin modules (primarily, we call PluginController.disablePluginModule to disable incompatible modules of other plugins), which sometimes provokes a deadlock identical tot he above:
2011-09-16 16:04:21,439 INFO [pool-6-thread-2] [atlassian.plugin.manager.DefaultPluginManager] disableDependentPlugins Found dependent enabled plugins for uninstalled plugin 'com.arsenale.plugins.lockpoint': []. Disabling... 2011-09-16 16:04:21,441 INFO [pool-6-thread-2] [atlassian.plugin.manager.DefaultPluginManager] updatePlugin Updating plugin 'com.arsenale.plugins.lockpoint' to 'com.arsenale.plugins.lockpoint' 2011-09-16 16:04:21,443 INFO [pool-6-thread-2] [atlassian.plugin.manager.DefaultPluginManager] notifyPluginDisabled Disabling com.arsenale.plugins.lockpoint 2011-09-16 16:04:23,133 ERROR [http-8090-5] [sf.hibernate.util.JDBCExceptionReporter] logExceptions Deadlock found when trying to get lock; try restarting transaction - referer: http://172.16.16.10:8090/plugins/servlet/upm | url: /rest/plugins/1.0/pending/008decc2-f4fb-421f-a4f0-6d05489257d7 | userName: admin 2011-09-16 16:04:23,134 ERROR [http-8090-5] [sf.hibernate.impl.SessionImpl] execute Could not synchronize database state with session - referer: http://172.16.16.10:8090/plugins/servlet/upm | url: /rest/plugins/1.0/pending/008decc2-f4fb-421f-a4f0-6d05489257d7 | userName: admin
If we call the above method too soon after our plugin gets initialized, we deadlock. If we wait a while before trying to disable the other plugin modules, everything works correctly. One correlating factor is that if we have not seen our own PluginInstallEvent by the time we try to call disablePluginModule(), we seem to deadlock. But once we've received that event, we seem to be okay.
In either case, the upgraded plugin is dead in the water after attempting the installation.
- relates to
-
-
- Closed
-
[CONFSERVER-23301] Database deadlock during upgrade of plugins
I've also seen this fail in 4.3.1 with errors similar to the following:
java.lang.RuntimeException: Hibernate operation: could not insert: [com.atlassian.confluence.setup.bandana.ConfluenceBandanaRecord#42]; SQL []; Duplicate entry '_GLOBAL-atlassian.confluence.plugin.counter' for key 'bandana_unique_key'; nested exception is com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry '_GLOBAL-atlassian.confluence.plugin.counter' for key 'bandana_unique_key'
This all seems to stem from a plugin manipulating the state of other plugins during plugin startup; Niraj's comments in CONF-21986 are particularly relevant (ie. suggesting that one not do that!).
Just to confirm, did you mean to say that you tried to reproduce it against Confluence 4.0 and not Confluence 3.2.1_01? We first discovered this issue for the first time with Confluence 4.0, and I do not believe the issue exists in 3.2.1_01.
Hi Scott,
We are still investigating this:
My name is Vincent and I am the escalation engineer for this ticket. We have tried to reproduce your issue on a fresh install on Confluence 3.2.1_01 without success.
It appears that the Plugin's data may of not updated properly in the upgrade process.
Could you please refer to this document to 'reset' the plugin to a pristine state.
This will all remove plugins and reset it (Warning: this will uninstall non bundled plugins - you can re-install these anytime afterwards).
Determine where your plugin is loaded. The usual options are:
1. In the database perform a 'DELETE from plugindata;' to remove all entries in the database.
2. The <confluence-home>/bundled-plugins folder (remove everything under here)
3. The <confluence-home>/plugin-cache folder (remove everything under here)
4. The <confluence-home>/plugins-osgi-cache folder (remove everything under here)
5. The <confluence-home>/plugins-temp folder (remove everything under here)
reboot confluence (the bundled plugins will then reinstall)
Could you please then clear the browser cache.
Then retest, and let us know how you go.
Regards
Vincent Choy
Thank you for taking the time to raise, comment or vote on this Bug. Currently this bug indicates that it only impacts a version of Confluence which is no longer in support, therefore we are closing this issue as Timed Out.
If this issue is still impacting you on a recent version please feel free to comment with the affected version. Any further details you may be able to provide regarding reproduction or impact of this issue may help us better address this issue.
Thanks again.
Regards,
Confluence Development