Updating apps in a multi-node cluster can randomly revert an app/module status in PLUGIN_SETTING to their default status across the cluster

XMLWordPrintable

    • 4
    • Severity 2 - Major
    • 2
    • RtB - Service ops & Tech entropy

      Issue Summary

      From Confluence 10.2.11+, apps and modules enabled status are persisted in PLUGIN_SETTING.

      Updating apps in Confluence Administration » Manage Apps in a multi-node Cluster can revert apps and modules back to their default state.

      Example:

      • Without an entry in PLUGIN_SETTING for namespace=plugin.manager.state.Map:
        • Apps revert to Enabled
        • Some internal modules such as confluence.macros.html:html-include revert to disabled

      Steps to Reproduce

      This issue can manifest itself in two different ways.

      App Update

      1. Install Confluence Data Center with at least 3 nodes (the more nodes, the more likely the issue will occur)
      2. Install several Apps that are older than the latest (so that the Update button appears). e.g.
      3. Once installed, disable all the Apps in Manage Apps screen
      4. Once all the apps are disabled, pick any outdated app (such as Confluence Cloud Migration Assistant) and press Update
      5. Perform a rolling restart of Confluence
      6. Check the app status in Confluence Administration » Manage Apps

      Simultaneous Node Restart

      In a multi-node environment, start up all nodes simultaneously or within the time it takes for the node to normally start up and be ready for service.

      Expected Results

      App Update

      All the installed third party apps (that were disabled) should remain as disabled.

      Simultaneous Node Restart

      All system plugins should be enabled after start up.

      Actual Results

      App Update

      1. Once an app is updated, it will broadcast for other nodes to update the app on their local nodes.
        • This occurs simultaneously across the cluster, leading to conflicts when nodes read from or write to PLUGIN_SETTING with random rows inadvertently removed
      2. Random apps are inadvertently re-enabled
        • If the issue does not occur, attempt another App update (the more nodes, the more likely the issue will occur)
      3. Checking atlassian-confluence.log will show either of these errors: 
        2026-09-04 06:55:23,766 WARN [nc.confluence.event-1] [confluence.impl.hibernate.ConfluenceHibernateTransactionManager] doCommit Commit failed. Rolling back. Error: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: delete from PLUGIN_SETTING where NAMESPACE=? and SETTING_KEY=?
2026-09-04 06:55:23,772 WARN [nc.confluence.event-1] [confluence.impl.hibernate.ConfluenceHibernateTransactionManager] doRollback Performing rollback. Transactions:\n  ->[org.springframework.data.jpa.repository.support.SimpleJpaRepository.deleteById]: PROPAGATION_REQUIRED,ISOLATION_DEFAULT (Session #2140378009)
2026-09-04 06:55:23,796 ERROR [nc.confluence.event-1] [confluence.impl.plugin.PluginSettingsPluginStateStore] save Exception when saving plugins state to the database. Saving to in memory state.
org.springframework.orm.hibernate5.HibernateOptimisticLockingFailureException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: delete from PLUGIN_SETTING where NAMESPACE=? and SETTING_KEY=?
	at org.springframework.orm.hibernate5.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:247)
	at org.springframework.orm.hibernate5.HibernateTransactionManager.convertHibernateAccessException(HibernateTransactionManager.java:788)
	at org.springframework.orm.hibernate5.HibernateTransactionManager.doCommit(HibernateTransactionManager.java:658)
..
	at jdk.proxy4/jdk.proxy4.$Proxy239.deleteById(Unknown Source)
	at com.atlassian.confluence.impl.settings.ConfluencePluginSettings.removeActual(ConfluencePluginSettings.java:202)
	at com.atlassian.sal.core.pluginsettings.AbstractStringPluginSettings.remove(AbstractStringPluginSettings.java:166)
	at com.atlassian.confluence.impl.settings.ConfluencePluginSettings.remove(ConfluencePluginSettings.java:192)
..
	at com.atlassian.plugin.manager.store.DelegatingPluginPersistentStateStore.save(DelegatingPluginPersistentStateStore.java:21)
	at com.atlassian.confluence.impl.plugin.CachingPluginPersistentStateStore.save(CachingPluginPersistentStateStore.java:32)
..
	at com.atlassian.confluence.impl.plugin.ConfluencePluginManager.processClusteredInstallEvent(ConfluencePluginManager.java:225)
	at com.atlassian.confluence.impl.plugin.ConfluencePluginManager.processPluginEvent(ConfluencePluginManager.java:187)
	at com.atlassian.confluence.impl.plugin.ConfluencePluginManager.onClusterEvent(ConfluencePluginManager.java:152)

        2026-09-04 08:27:59,366 WARN [nc.confluence.event-1] [confluence.impl.plugin.PluginSettingsPluginStateStore] lambda$load$1 Unable to deserialize plugin state entry for key hash d18349d2ec626b530312c5ee7a3039ca. Skipping entry. Error: Cannot invoke \"com.atlassian.confluence.impl.plugin.PluginSettingsPluginStateStore$PluginStateStoreEntry.getKey()\" because \"stateEntry\" is null

      Simultaneous Node Restart

      Within the application logs, an entry like the following will be recorded indicating that Confluence couldn't persist the state of a given plugin most likely due to a concurrent race condition:
       

      2026-07-21 01:05:05,084 ERROR [Catalina-utility-1] [confluence.impl.plugin.PluginSettingsPluginStateStore] save Exception when saving plugins state to the database. Saving to in memory state. org.springframework.orm.hibernate5.HibernateOptimisticLockingFailureException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: delete from PLUGIN_SETTING where NAMESPACE=? and SETTING_KEY=?  at org.springframework.orm.hibernate5.SessionFactoryUtils.convertHibernateAccessException(SessionFactoryUtils.java:247)  ...  Caused by: org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1; statement executed: delete from PLUGIN_SETTING where NAMESPACE=? and SETTING_KEY=?

      As a result of not being able to persist the plugin state, Confluence then disables the affected plugin:
       

      2026-07-21 01:05:05,104 ERROR [Catalina-utility-1] [atlassian.plugin.manager.DefaultPluginManager] lambda$enableConfiguredPluginModule$32 There was an error loading the descriptor 'mention-hipchat-notification-template' of plugin 'com.atlassian.confluence.plugins.confluence-mentions-plugin'. Disabling. org.springframework.transaction.UnexpectedRollbackException: Transaction rolled back because it has been marked as rollback-only  at org.springframework.transaction.support.AbstractPlatformTransactionManager.processRollback(AbstractPlatformTransactionManager.java:938)  at org.springframework.transaction.support.AbstractPlatformTransactionManager.commit(AbstractPlatformTransactionManager.java:754) ...  at jdk.proxy4/jdk.proxy4.$Proxy355.enablePluginModule(Unknown Source)  

      This manifestation of the issue has been seen to affect the following plugins:

      • confluence-notifications-batch-plugin
      • confluence-mentions-plugin
      • confluence-app-monitoring
      • share-page
      • confluence-inline-tasks
      • confluence-content-notifications-plugin
      • confluence-inline-comments

      Workarounds

      App Update

      1. Scale the Cluster down to one node before updating Apps.
      2. Once all the required apps are updated, scale Confluence back to the required number of nodes

      Simultaneous Node Restart

      Stagger start the nodes by allowing each node several minutes of time to complete it's individual start up.

        1. AppsAfterRestart.png
          AppsAfterRestart.png
          686 kB
        2. AppsBeforeRestart.png
          AppsBeforeRestart.png
          709 kB
        3. AppUpdateAcrossCluster.png
          AppUpdateAcrossCluster.png
          787 kB

              Assignee:
              agawron
              Reporter:
              Eric L
              Votes:
              3 Vote for this issue
              Watchers:
              12 Start watching this issue

                Created:
                Updated:
                Resolved: