-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Highest
-
Affects Version/s: 10.2.10
-
Component/s: Content - Blueprints / Templates
-
1
-
Severity 2 - Major
Problem
When upgrading Confluence with a large amount of users (50k+), the upgrade will hang for an extended period of time, with the Catalina-utility-1 thread consuming 100% CPU.
Environment
- Confluence 9.2.15 > upgrading to 10.2.10
Steps to Reproduce
In Confluence 9.2.15:
- Create a large amount of users (e.g 50k)
- I have included the generateusers.py
python script for ease of generation (it will run for ~2minutes or so)
- I have included the generateusers.py
- Insert the below users into com.atlassian.confluence.plugins.createcontent.impl.DefaultUserStorageService (contained within BANDANA)
- I have included the exact SQL UPDATE statement for ease of insertion of the 50k users - insertusers.sql
- I have included the exact SQL UPDATE statement for ease of insertion of the 50k users - insertusers.sql
- Shut down Confluence, and upgrade to 10.2.10
Expected Results
The upgrade will progress as expected, and will complete within a reasonable amount of time
Actual Results
Accessing Confluence in the browser will hang for an extended period of time
atlassian-confluence.log will contain only:
2026-04-20 10:13:31,750 INFO [Catalina-utility-1] [agent.service.impl.DefaultStatisticsService] onPluginEnabled CCMA Plugin enabled, calculating server statistics if needed 2026-04-20 10:13:31,815 INFO [Catalina-utility-1] [agent.service.impl.MigrationSpaceStatusService] onPluginEnabled CCMA Plugin enabled, populating migration space status 2026-04-20 10:13:31,818 INFO [Catalina-utility-1] [atlassian.migration.agent.PluginLauncher] onLifecycleEvent Life cycle event PLUGIN_ENABLED received 2026-04-20 10:13:31,827 INFO [SiteSummaryCalculator:thread-1] [agent.service.impl.DefaultStatisticsService] calculateServerStats Starting to Calculate server statistics 2026-04-20 10:13:31,827 INFO [SiteSummaryCalculator:thread-1] [agent.service.impl.DefaultStatisticsService] calculateServerStats Completed Instance Summary Calculation 2026-04-20 10:13:31,827 INFO [CCMA-MigrationSpaceStatusService-pool-17-thread-1] [agent.service.impl.MigrationSpaceStatusService] populateMigrationSpaceStatuses Populating initial migration space statuses with size: 0 2026-04-20 10:13:31,838 INFO [SiteSummaryCalculator:thread-1] [agent.service.impl.DefaultStatisticsService] calculateServerStats Completed Content Summary Calculation 2026-04-20 10:13:31,975 INFO [Catalina-utility-1] [atlassian.plugin.manager.DefaultPluginManager] logTime Plugin system lateStartup ended 2026-04-20 10:13:32,070 INFO [Catalina-utility-1] [atlassian.secrets.service.DefaultSecretServiceManagement] reloadConfiguration Re-loading SecretService configuration 2026-04-20 10:13:32,071 INFO [Catalina-utility-1] [atlassian.secrets.service.DefaultSecretService] reconfigure Re-configuring SecretService to use new default backend: defaultAES
there will be no additional logging for an extended period of time.
Taking thread dumps of Confluence will show 99.9% CPU busy only on Catalina-utility-1 thread, with the below stack trace:
catalina.out
"Catalina-utility-1" #24 [42243] prio=1 os_prio=31 cpu=109558.70ms elapsed=145.28s tid=0x0000000940c15500 nid=42243 runnable [0x000000016e08b000] java.lang.Thread.State: RUNNABLE at org.hibernate.event.internal.ProxyVisitor.processEntity(ProxyVisitor.java:31) at org.hibernate.event.internal.AbstractVisitor.processValue(AbstractVisitor.java:107) at org.hibernate.event.internal.WrapVisitor.processValue(WrapVisitor.java:132) at org.hibernate.event.internal.AbstractVisitor.processEntityPropertyValues(AbstractVisitor.java:59) at org.hibernate.event.internal.DefaultFlushEntityEventListener.wrapCollections(DefaultFlushEntityEventListener.java:237) at org.hibernate.event.internal.DefaultFlushEntityEventListener.onFlushEntity(DefaultFlushEntityEventListener.java:172) at org.hibernate.event.internal.AbstractFlushingEventListener$$Lambda/0x00000070038d5288.accept(Unknown Source) at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:107) at org.hibernate.event.internal.AbstractFlushingEventListener.flushEntities(AbstractFlushingEventListener.java:229) at org.hibernate.event.internal.AbstractFlushingEventListener.flushEverythingToExecutions(AbstractFlushingEventListener.java:93) at org.hibernate.event.internal.DefaultAutoFlushEventListener.onAutoFlush(DefaultAutoFlushEventListener.java:50) at org.hibernate.internal.SessionImpl$$Lambda/0x00000070027710d0.accept(Unknown Source) at org.hibernate.event.service.internal.EventListenerGroupImpl.fireEventOnEachListener(EventListenerGroupImpl.java:107) at org.hibernate.internal.SessionImpl.autoFlushIfRequired(SessionImpl.java:1372) at org.hibernate.internal.SessionImpl.list(SessionImpl.java:1452) at org.hibernate.query.internal.AbstractProducedQuery.doList(AbstractProducedQuery.java:1649) at org.hibernate.query.internal.AbstractProducedQuery.list(AbstractProducedQuery.java:1617) at org.hibernate.query.internal.AbstractProducedQuery.uniqueResult(AbstractProducedQuery.java:1659) at org.hibernate.query.internal.AbstractProducedQuery.uniqueResultOptional(AbstractProducedQuery.java:1610) at com.atlassian.confluence.impl.user.preferences.dao.UserPreferencesDaoImpl.findPropertySetItem(UserPreferencesDaoImpl.java:109) at com.atlassian.confluence.impl.user.preferences.dao.UserPreferencesDaoImpl.setUserPreference(UserPreferencesDaoImpl.java:89) at com.atlassian.confluence.impl.user.preferences.dao.UserPreferencesDaoImpl.setUserPreference(UserPreferencesDaoImpl.java:79) ... at jdk.proxy4.$Proxy237.setUserPreference(jdk.proxy4/Unknown Source) at com.atlassian.confluence.impl.user.preferences.dao.CachingUserPreferencesDao.setUserPreference(CachingUserPreferencesDao.java:82) at com.atlassian.confluence.impl.user.preferences.UserPreferencesImpl.setBoolean(UserPreferencesImpl.java:62) at com.atlassian.confluence.plugins.createcontent.upgrade.BlueprintDiscoveryUpgradeTask.lambda$doUpgrade$0(BlueprintDiscoveryUpgradeTask.java:104) at com.atlassian.confluence.plugins.createcontent.upgrade.BlueprintDiscoveryUpgradeTask$$Lambda/0x0000007003b82d58.doInTransaction(Unknown Source) at com.atlassian.sal.core.transaction.HostContextTransactionTemplate$1.doInTransaction(HostContextTransactionTemplate.java:21) at com.atlassian.sal.spring.component.SpringHostContextAccessor.lambda$doInTransaction$0(SpringHostContextAccessor.java:72) at com.atlassian.sal.spring.component.SpringHostContextAccessor$$Lambda/0x0000007003acfb70.doInTransaction(Unknown Source) ... at com.atlassian.sal.spring.component.SpringHostContextAccessor.doInTransaction(SpringHostContextAccessor.java:70) ... at jdk.proxy4.$Proxy392.execute(jdk.proxy4/Unknown Source) ... at jdk.proxy50.$Proxy1871.execute(jdk.proxy50/Unknown Source) at com.atlassian.confluence.plugins.createcontent.upgrade.BlueprintDiscoveryUpgradeTask.lambda$doUpgrade$1(BlueprintDiscoveryUpgradeTask.java:97) at com.atlassian.confluence.plugins.createcontent.upgrade.BlueprintDiscoveryUpgradeTask$$Lambda/0x0000007003b82b20.accept(Unknown Source) at java.lang.Iterable.forEach(java.base@21.0.8/Iterable.java:75) at com.atlassian.confluence.plugins.createcontent.upgrade.BlueprintDiscoveryUpgradeTask.doUpgrade(BlueprintDiscoveryUpgradeTask.java:96)
Eventually, the upgrade will complete (for 40k users, this took ~4 hours)
Workaround
Wait for the upgrade to complete (this can take several hours depending on the number of users)
Notes
N/A