Issue Summary

When a Confluence DC instance is being upgraded, the upgrade process starts by tagging CONFVERSION.VERSIONTAG with something like:

confversionid buildnumber installdate         versiontag               creationdate lastmoddate finalized 
------------- ----------- ------------------- ------------------------ ------------ ----------- --------- 
432701441     7901        2022-05-30 10:16:04 lock_for_upgrade_to_8803 (null)       (null)      N         

Example:

2022-05-30 10:26:56,046 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 7.16.3 [build 8803 based on commit hash 0bc9f9669a40f99ddafa36d34382eab10dd5c59e] - synchrony version 4.0.0-master-bfb398fb
...
2022-05-30 10:28:02,912 DEBUG [Catalina-utility-1] [confluence.upgrade.impl.DefaultUpgradeManager] tryAcquireDatabaseUpgradesLock Successfully tagged build number 7901 with 'lock_for_upgrade_to_8803', database upgrades are permitted
...
2022-05-30 10:28:09,030 DEBUG [Catalina-utility-1] [confluence.upgrade.recovery.ConfluenceProgressMonitor] logTask Starting TABLE_DATA BODYCONTENT

If Confluence is now restarted (whilst the lock_for_upgrade_to_XXXX is still in place), Confluence will then deem the Confluence upgrade as "successful", despite skipping all the Upgrade Tasks .

This is reproducible on Data Center: yes

Steps to Reproduce

1. Install an old version of Confluence DC (eg. Confluence 6.13.8)
2. Ensure that it is fully installed and running
3. Now, upgrade Confluence to say, Confluence 7.16.3
4. Before starting Confluence 7.16.3, perform the following:
   • Enable the following DEBUG in <confluence-install>/confluence/WEB-INF/classes/log4j.properties:

     log4j.logger.com.atlassian.confluence.internal.upgrade=DEBUG
     log4j.logger.com.atlassian.confluence.upgrade=DEBUG
     

5. Start Confluence 7.16.3 and the moment it gets to the step of backing up tables, just kill Confluence
   • Alternate to the above step, you can simulate the condition by simply updating the CONFVERSION table with:

     update confversion set versiontag = 'lock_for_upgrade_to_8803' where confversionid in (select max(confversionid) from confversion);
     

   • For the above "SQL simulation", refer to Build Number for the target Confluence version
6. Start Confluence 7.16.3 again whilst the lock_for_upgrade_to_8803 is in place and tail the atlassian-confluence.log

Expected Results

• The upgrade should not be deemed completed "successful".
• CONFVERSION should not be updated to the latest internal build number.
  • Restarting the same node or starting a second node would subsequently show:

    Confluence cluster node will not start up because the build number in the database [7502] doesn’t match either the application build number [8703] or the home directory build number [8703].
    

Actual Results

The upgrade will skip over the Upgrade Tasks as shown in the logs:

2022-05-30 10:32:03,756 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 7.16.3 [build 8803 based on commit hash 0bc9f9669a40f99ddafa36d34382eab10dd5c59e] - synchrony version 4.0.0-master-bfb398fb
...
2022-05-30 10:33:08,635 DEBUG [Catalina-utility-1] [confluence.upgrade.impl.DefaultUpgradeManager] tryAcquireDatabaseUpgradesLock Cluster upgrade lock acquired. Attempting to tag build number 7901
2022-05-30 10:33:08,674 DEBUG [Catalina-utility-1] [confluence.upgrade.impl.DefaultUpgradeManager] tryAcquireDatabaseUpgradesLock Failed to tag build number 7901 with 'lock_for_upgrade_to_8803'
2022-05-30 10:33:08,733 DEBUG [Catalina-utility-1] [atlassian.confluence.upgrade.AbstractUpgradeManager] executeUpgradeTask Skipping SCHEMA_VALIDATION phase for database upgrade task for build number 8202, 'synchronyEvictionEventsPreSchemaUpgradeTask'
2022-05-30 10:33:08,740 DEBUG [Catalina-utility-1] [atlassian.confluence.upgrade.AbstractUpgradeManager] executeUpgradeTask Skipping SCHEMA_VALIDATION phase for database upgrade task for build number 8202, 'synchronyEvictionSnapshotsPreSchemaUpgradeTask'
2022-05-30 10:33:21,153 DEBUG [Catalina-utility-1] [atlassian.confluence.upgrade.AbstractUpgradeManager] executeUpgradeTask Skipping SCHEMA_VALIDATION phase for database upgrade task for build number 8201, 'dropTrackBackAndReferrerTablesUpgradeTask'
2022-05-30 10:33:21,159 DEBUG [Catalina-utility-1] [atlassian.confluence.upgrade.AbstractUpgradeManager] executeUpgradeTask Skipping SCHEMA_VALIDATION phase for database upgrade task for build number 8202, 'synchronyEvictionEventsPostSchemaUpgradeTask'
2022-05-30 10:33:21,160 DEBUG [Catalina-utility-1] [atlassian.confluence.upgrade.AbstractUpgradeManager] executeUpgradeTask Skipping SCHEMA_VALIDATION phase for database upgrade task for build number 8202, 'synchronyEvictionSnapshotsPostSchemaUpgradeTask'
2022-05-30 10:33:21,169 DEBUG [Catalina-utility-1] [atlassian.confluence.upgrade.AbstractUpgradeManager] executeUpgradeTask Skipping SCHEMA_VALIDATION phase for database upgrade task for build number 8301, 'addMissingOsPropertyIndexUpgradeTask'
2022-05-30 10:33:21,174 DEBUG [Catalina-utility-1] [atlassian.confluence.upgrade.AbstractUpgradeManager] executeUpgradeTask Skipping SCHEMA_VALIDATION phase for database upgrade task for build number 8507, 'denormalisedSpacePermissionsUpgradeTask'
2022-05-30 10:33:21,174 DEBUG [Catalina-utility-1] [atlassian.confluence.upgrade.AbstractUpgradeManager] executeUpgradeTask Skipping SCHEMA_VALIDATION phase for database upgrade task for build number 8701, 'denormalisedContentPermissionsUpgradeTask'
2022-05-30 10:33:21,174 DEBUG [Catalina-utility-1] [atlassian.confluence.upgrade.AbstractUpgradeManager] executeUpgradeTask Skipping SCHEMA_VALIDATION phase for database upgrade task for build number 8700, 'embeddedCrowdAddGroupExternalIdTask'
2022-05-30 10:33:21,174 DEBUG [Catalina-utility-1] [atlassian.confluence.upgrade.AbstractUpgradeManager] executeUpgradeTask Skipping SCHEMA_VALIDATION phase for database upgrade task for build number 8703, 'denormalisedContentPermissionsTriggersUpgradeTask'
...
2022-05-30 10:35:03,927 INFO [Catalina-utility-1] [com.atlassian.confluence.lifecycle] init Confluence is ready to serve

CONFVERSION is also updated to the new build number

Depending which version Confluence is upgraded from (e.g. from Confluence 6.13), manually checking the database schema for a few upgrade tasks:

• dropTrackBackAndReferrerTablesUpgradeTask
  This should have run

  2022-05-30 11:13:11,379 INFO [Catalina-utility-1] [confluence.upgrade.ddl.HibernateDdlExecutor] executeDdlStatements Executing DDL: drop table TRACKBACKLINKS
  2022-05-30 11:13:11,389 INFO [Catalina-utility-1] [confluence.upgrade.ddl.HibernateDdlExecutor] executeDdlStatements Executing DDL: drop table EXTRNLNKS
  

  • When checking for the above two tables, they are still there :

    schemaname tablename      
    ---------- -------------- 
    public     extrnlnks      
    public     trackbacklinks 
    

• denormalisedContentPermissionsTriggersUpgradeTask
  • There should be 13 triggers created:

    table_name       trigger_name                                      
    ---------------- ------------------------------------------------- 
    spaces           denormalised_space_trigger_on_update              
    spaces           denormalised_space_trigger_on_insert              
    spaces           denormalised_space_trigger_on_delete              
    spacepermissions denormalised_space_permission_trigger_on_update   
    spacepermissions denormalised_space_permission_trigger_on_insert   
    spacepermissions denormalised_space_permission_trigger_on_delete   
    content          denormalised_content_trigger_on_update            
    content          denormalised_content_trigger_on_insert            
    content          denormalised_content_trigger_on_delete            
    content_perm     denormalised_content_permission_trigger_on_update 
    content_perm     denormalised_content_permission_trigger_on_insert 
    content_perm     denormalised_content_permission_trigger_on_delete 
    content_perm_set denormalised_content_perm_set_trigger_on_delete   
    

  • When checking for triggers, there are none created post upgrade :

    SELECT  event_object_table AS table_name ,trigger_name FROM information_schema.triggers;
    
    0 row(s) returned
    

• synchronyEvictionEventsPreSchemaUpgradeTask
  • This upgrade task should have dropped EVENTS and SNAPSHOTS and recreate table with new CONTENTID column:
    This should have run

    2022-05-30 11:13:09,375 DEBUG [Catalina-utility-1] [atlassian.confluence.upgrade.AbstractUpgradeManager] executeUpgradeTask Executing SCHEMA_UPGRADE phase for database upgrade task for build number 8202, 'synchronyEvictionEventsPreSchemaUpgradeTask'
    2022-05-30 11:13:09,379 INFO [Catalina-utility-1] [confluence.upgrade.ddl.HibernateDdlExecutor] executeDdlStatements Executing DDL: DROP TABLE IF EXISTS "EVENTS"
    2022-05-30 11:13:09,466 DEBUG [Catalina-utility-1] [atlassian.confluence.upgrade.AbstractUpgradeManager] executeUpgradeTask Executing SCHEMA_UPGRADE phase for database upgrade task for build number 8202, 'synchronyEvictionSnapshotsPreSchemaUpgradeTask'
    2022-05-30 11:13:09,467 INFO [Catalina-utility-1] [confluence.upgrade.ddl.HibernateDdlExecutor] executeDdlStatements Executing DDL: DROP TABLE IF EXISTS "SNAPSHOTS"
    

  • Checking EVENTS and SNAPSHOTS post upgrade:
    • Neither of these tables have the new contentid column

The above is just a sample set of the upgrade tasks that were confirmed not to have run.

Workaround

Roll back to pre-upgrade and perform the upgrade again without restarting Confluence once it has started.

As best practice, Atlassian recommends performing testing upgrades in a Staging Environment before upgrading the Production instance.