Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-36430

On certain configurations, the attachments upgrade task very slow or hangs when upgrading to 5.7

      This is an umbrella issue to investigate why the Confluence upgrade to 5.7 takes a long amount of time with certain database configurations or in some rare cases even hangs.

      Here is an example log message:

      2015-02-05 01:35:26,704 INFO [localhost-startStop-1] [upgrade.upgradetask.attachmentceo.AttachmentIdClashResolver] resolveAttachmentIdClashes No ID collisions were found between the ATTACHMENTS and CONTENT tables.
      2015-02-05 01:35:26,717 INFO [localhost-startStop-1] [upgrade.upgradetask.attachmentceo.AttachmentToContentMigrator] doMigration Beginning migration of 6644 attachment entries..
      

      This appears to be affecting SQL Server and Oracle databases that already have latency issues.

      Workaround for SQL Server

      • SQL Server is known to have this issue when a connection validation query is enabled or when C3P0 (direct JDBC) connection pooling is used.
      • There is a bug (CONF-36484) where the preferredTestQuery passed to C3P0 is ignored causing it to issue a "list tables" query which locks up if run while the upgrade task is running.
      • The solution is to switch to using a datasource connection with "select 1" as the connection test query.

            [CONFSERVER-36430] On certain configurations, the attachments upgrade task very slow or hangs when upgrading to 5.7

            This particular upgrade task does try and be transactional and resumable, but that is not always the case with al upgrade tasks, so as a general rule it's better to restore. Glad it worked out for you though!

            Niraj Bhawnani added a comment - This particular upgrade task does try and be transactional and resumable, but that is not always the case with al upgrade tasks, so as a general rule it's better to restore. Glad it worked out for you though!

            Thank you for the feedback.

            Before I decided to restore from our VM and DB backup, I decided to restart the upgrade just to see if it would work. What the heck, right? Ours hung on the following line:

            2015-10-23 01:43:15,093 INFO [localhost-startStop-1] [upgrade.upgradetask.attachmentceo.AttachmentToContentMigrator] lambda$doMigration$435 Migrated 16273 (22712 of 108068) attachment entries to the content table in 16s.
            

            I performed the instructions https://confluence.atlassian.com/doc/configuring-a-sql-server-datasource-in-apache-tomcat-218269283.html and restarted Confluence and it went through all the steps this time and succeeded.

            I know it was not recommended and your mileage may vary but it seemed to work. We're also going to move to Oracle to avoid stuff like thisl

            David Hergert (PAYX) added a comment - Thank you for the feedback. Before I decided to restore from our VM and DB backup, I decided to restart the upgrade just to see if it would work. What the heck, right? Ours hung on the following line: 2015-10-23 01:43:15,093 INFO [localhost-startStop-1] [upgrade.upgradetask.attachmentceo.AttachmentToContentMigrator] lambda$doMigration$435 Migrated 16273 (22712 of 108068) attachment entries to the content table in 16s. I performed the instructions https://confluence.atlassian.com/doc/configuring-a-sql-server-datasource-in-apache-tomcat-218269283.html and restarted Confluence and it went through all the steps this time and succeeded. I know it was not recommended and your mileage may vary but it seemed to work. We're also going to move to Oracle to avoid stuff like thisl

            Hi dhergert1, sorry to hear you've hit this as well. The upgrade won't recover if you have to stop it midway through, and the only way to 'roll back' is to restore your pre-upgrade backups, and start again. This is because the changes that are made to the database as part of an upgrade are usually not compatible with the code in older versions, so you can't start an older version of Confluence over data that has been partially (or completely) upgraded. Likewise, if the upgrade hasn't completely finished the database will be in an inconsistent state, and will cause any later attempts at upgrade to fail. Restoring backups, or waiting, are your only two options. Hope this information helps!

            Denise Unterwurzacher [Atlassian] (Inactive) added a comment - Hi dhergert1 , sorry to hear you've hit this as well. The upgrade won't recover if you have to stop it midway through, and the only way to 'roll back' is to restore your pre-upgrade backups, and start again. This is because the changes that are made to the database as part of an upgrade are usually not compatible with the code in older versions, so you can't start an older version of Confluence over data that has been partially (or completely) upgraded. Likewise, if the upgrade hasn't completely finished the database will be in an inconsistent state, and will cause any later attempts at upgrade to fail. Restoring backups, or waiting, are your only two options. Hope this information helps!

            This happened to us as well, what do we do with an upgrade that is hung mid-upgrade? Is a roll-back the only option or can it be continued after moving to a different data source?

            David Hergert (PAYX) added a comment - This happened to us as well, what do we do with an upgrade that is hung mid-upgrade? Is a roll-back the only option or can it be continued after moving to a different data source?

            I can't access the database while it upgrades, because the lock request times out, so I assume, the upgrade has put the database in single user mode and now it blocks its own threads.

            Nabil Sayegh added a comment - I can't access the database while it upgrades, because the lock request times out, so I assume, the upgrade has put the database in single user mode and now it blocks its own threads.

            We face the exact problem when upgrading from 5.6.6 to 5.8.14.
            We're using a confluence.xml.cfg datasource:

            <property name="hibernate.c3p0.acquire_increment">1</property>
            <property name="hibernate.c3p0.idle_test_period">100</property>
            <property name="hibernate.c3p0.max_size">30</property>
            <property name="hibernate.c3p0.max_statements">0</property>
            <property name="hibernate.c3p0.min_size">0</property>
            <property name="hibernate.c3p0.preferredTestQuery">select 1</property>
            <property name="hibernate.c3p0.timeout">30</property>
            <property name="hibernate.c3p0.validate">true</property>
            <property name="hibernate.connection.driver_class">net.sourceforge.jtds.jdbc.Driver</property>
            <property name="hibernate.connection.isolation">2</property>
            <property name="hibernate.connection.password">xxxxxxxxxx</property>
            <property name="hibernate.connection.url">jdbc:jtds:sqlserver://db.example.com:1234/confluence</property>
            <property name="hibernate.connection.username">confluence</property>
            <property name="hibernate.database.lower_non_ascii_supported">true</property>
            <property name="hibernate.dialect">net.sf.hibernate.dialect.SQLServerIntlDialect</property>
            <property name="hibernate.setup">true</property>

            2015-10-14 13:40:01,775 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 5.8.14 [build 5994 based on commit hash 3a465f7321af83b92e8f7149d515f72cee083e87]
            2015-10-14 13:40:04,205 INFO [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization started
            2015-10-14 13:40:11,929 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] <init> Loading EhCache cache manager
            2015-10-14 13:40:15,621 WARN [localhost-startStop-1] [org.springframework.beans.TypeConverterDelegate] findDefaultEditor PropertyEditor [com.sun.beans.editors.EnumEditor] found through deprecated global PropertyEditorManager fallback - consider using a more isolated form of registration, e.g. on the BeanWrapper/BeanFactory!
            2015-10-14 13:40:30,326 INFO [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization completed in 26120 ms
            2015-10-14 13:40:55,448 INFO [localhost-startStop-1] [upgrade.upgradetask.attachmentceo.AttachmentIdClashResolver] resolveAttachmentIdClashes No ID collisions were found between the ATTACHMENTS and CONTENT tables.
            2015-10-14 13:40:55,570 INFO [localhost-startStop-1] [upgrade.upgradetask.attachmentceo.AttachmentToContentMigrator] doMigration Beginning migration of 96032 attachment entries...

            Nabil Sayegh added a comment - We face the exact problem when upgrading from 5.6.6 to 5.8.14. We're using a confluence.xml.cfg datasource: <property name="hibernate.c3p0.acquire_increment">1</property> <property name="hibernate.c3p0.idle_test_period">100</property> <property name="hibernate.c3p0.max_size">30</property> <property name="hibernate.c3p0.max_statements">0</property> <property name="hibernate.c3p0.min_size">0</property> <property name="hibernate.c3p0.preferredTestQuery">select 1</property> <property name="hibernate.c3p0.timeout">30</property> <property name="hibernate.c3p0.validate">true</property> <property name="hibernate.connection.driver_class">net.sourceforge.jtds.jdbc.Driver</property> <property name="hibernate.connection.isolation">2</property> <property name="hibernate.connection.password">xxxxxxxxxx</property> <property name="hibernate.connection.url">jdbc:jtds:sqlserver://db.example.com:1234/confluence</property> <property name="hibernate.connection.username">confluence</property> <property name="hibernate.database.lower_non_ascii_supported">true</property> <property name="hibernate.dialect">net.sf.hibernate.dialect.SQLServerIntlDialect</property> <property name="hibernate.setup">true</property> 2015-10-14 13:40:01,775 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] contextInitialized Starting Confluence 5.8.14 [build 5994 based on commit hash 3a465f7321af83b92e8f7149d515f72cee083e87] 2015-10-14 13:40:04,205 INFO [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization started 2015-10-14 13:40:11,929 INFO [localhost-startStop-1] [com.atlassian.confluence.lifecycle] <init> Loading EhCache cache manager 2015-10-14 13:40:15,621 WARN [localhost-startStop-1] [org.springframework.beans.TypeConverterDelegate] findDefaultEditor PropertyEditor [com.sun.beans.editors.EnumEditor] found through deprecated global PropertyEditorManager fallback - consider using a more isolated form of registration, e.g. on the BeanWrapper/BeanFactory! 2015-10-14 13:40:30,326 INFO [localhost-startStop-1] [springframework.web.context.ContextLoader] initWebApplicationContext Root WebApplicationContext: initialization completed in 26120 ms 2015-10-14 13:40:55,448 INFO [localhost-startStop-1] [upgrade.upgradetask.attachmentceo.AttachmentIdClashResolver] resolveAttachmentIdClashes No ID collisions were found between the ATTACHMENTS and CONTENT tables. 2015-10-14 13:40:55,570 INFO [localhost-startStop-1] [upgrade.upgradetask.attachmentceo.AttachmentToContentMigrator] doMigration Beginning migration of 96032 attachment entries...

            We faced this issue while upgrading from 5.6.5 to 5.8.8 on PROD. We had to rollback changes. Changing the Data source to Tomcat resolved the problem on QA Upgrade.

            The link to do that is https://confluence.atlassian.com/display/DOC/Configuring+a+SQL+Server+Datasource+in+Apache+Tomcat

            We still have to upgrade the PROD environment which is scheduled on Friday. Will update how that goes.

            Shraddha Jaiswal added a comment - We faced this issue while upgrading from 5.6.5 to 5.8.8 on PROD. We had to rollback changes. Changing the Data source to Tomcat resolved the problem on QA Upgrade. The link to do that is https://confluence.atlassian.com/display/DOC/Configuring+a+SQL+Server+Datasource+in+Apache+Tomcat We still have to upgrade the PROD environment which is scheduled on Friday. Will update how that goes.

            Tom Jackson added a comment - - edited

            We are seeing this while upgrading from 5.4.4 to 5.8.8, with a SQL Server DB using direct JDBC.

            I will try switching to a datasource connection

            update
            switching to datasource did in fact allow the upgrade to procede

            Tom Jackson added a comment - - edited We are seeing this while upgrading from 5.4.4 to 5.8.8, with a SQL Server DB using direct JDBC. I will try switching to a datasource connection update switching to datasource did in fact allow the upgrade to procede

            April,

            I'm one among the many reporters of this bug and we did face the issue during the to 5.7 and I did revert back to my previous db configuration without much difference.

            Can anyone comment on performance considerations with this configuration?

            IMO , 5.7 was hopeless for us in terms of performance and things have improved since we moved up into 5.8.4

            In other words, with a larger installation, is it appropriate to leave this config in place, or would it be best to revert after the upgrade is complete?

            Should be harmless but I would suggest good to revert back after the upgrade is complete.

            Madhusudhan Matrubai added a comment - April, I'm one among the many reporters of this bug and we did face the issue during the to 5.7 and I did revert back to my previous db configuration without much difference. Can anyone comment on performance considerations with this configuration? IMO , 5.7 was hopeless for us in terms of performance and things have improved since we moved up into 5.8.4 In other words, with a larger installation, is it appropriate to leave this config in place, or would it be best to revert after the upgrade is complete? Should be harmless but I would suggest good to revert back after the upgrade is complete.

            Hi April,

            we have:
            Total Spaces 370
            Site Spaces 218
            Personal Spaces 152
            Content (All Versions) 124713
            Content (Current Versions) 53097

            And with no change since confluence 3.x in database config, the upgrade from 5.6 to 5.7 was as fast as everytime before.

            Regards,
            Dirk

            Dirk Bromberg added a comment - Hi April, we have: Total Spaces 370 Site Spaces 218 Personal Spaces 152 Content (All Versions) 124713 Content (Current Versions) 53097 And with no change since confluence 3.x in database config, the upgrade from 5.6 to 5.7 was as fast as everytime before. Regards, Dirk

              nbhawnani Niraj Bhawnani
              mreis Mitermayer Reis
              Affected customers:
              9 This affects my team
              Watchers:
              42 Start watching this issue

                Created:
                Updated:
                Resolved: