Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-9888

Attachment migration from filesystem to database or vice versa cannot move large numbers of attachments

      I am running the migration from file-system to database storage of attachments (so I can run Massive). In my trial run, I have 13 GB of attachments. It seems that it is not committing the files to the database until all are copied there, which is filling up our temp tablespace on our testing db. (our temp space was 8GB). My question is, am I correct in saying that confluence will import all files then commit, or does it commit after each file or group of files? This could be a big deal when I import our production data if it is all at one bang. (i.e. What would happen if we had 1TB of attachments?)

            [CONFSERVER-9888] Attachment migration from filesystem to database or vice versa cannot move large numbers of attachments

            Hi Qian, I'd very much recommend you hold off on progressing with clustered, we'll be releasing a totally new offering with 5.6 named Confluence Data Center which includes a rearchitected clustering solutions, which (amongst many other things) no longer requires attachments to be stored in the database.

            John Masson added a comment - Hi Qian, I'd very much recommend you hold off on progressing with clustered, we'll be releasing a totally new offering with 5.6 named Confluence Data Center which includes a rearchitected clustering solutions, which (amongst many other things) no longer requires attachments to be stored in the database.

            Qian Zhao added a comment -

            I'm using the cluster version, and the attachment migration to DB is required according to the document. If the attachments need to be moved back to the file system in the future, it'll pose a problem for us as we have two servers running Confluence and one database.

            Qian Zhao added a comment - I'm using the cluster version, and the attachment migration to DB is required according to the document. If the attachments need to be moved back to the file system in the future, it'll pose a problem for us as we have two servers running Confluence and one database.

            Hi Qian.

            Storing attachments in the database has been deprecated in Confluence 5.5. The capability will be removed in some future version of Confluence, although we will need to provide a robust mechanism for customers using database storage to migrate to the file system.

            Paul Curren added a comment - Hi Qian. Storing attachments in the database has been deprecated in Confluence 5.5. The capability will be removed in some future version of Confluence, although we will need to provide a robust mechanism for customers using database storage to migrate to the file system.

            Qian Zhao added a comment -

            I am having the same problem when migrating attachment from file system to database on our 5.2.5 cluster.

            2014-07-16 09:40:37,769 ERROR [Long running task: Attachment data migration] [confluence.util.longrunning.AttachmentMigrationLongRunningTask] runInternal Could not create Oracle LOB; nested exception is java.io.IOException: ORA-01652: unable to extend temp segment by 128 in tablespace TEMP

            – referer: http://www.xxx.com/admin/doeditattachmentstorage.action | url: /admin/doattachmentmigration.action | userName: xxx | action: doattachmentmigration
            org.springframework.dao.DataAccessResourceFailureException: Could not create Oracle LOB; nested exception is java.io.IOException: ORA-01652: unable to extend temp segment by 128 in tablespace TEMP

            Before the above error, I also got some warnings like this:

            2014-07-16 09:38:29,376 WARN [Long running task: Attachment data migration] [persistence.dao.hibernate.AbstractHibernateAttachmentDao$IntraHibernateAttachmentCopier] saveAttachmentData Attachment size of Attachment: RAM Upgrade Process v.3 (3473418) yge is greater than 0, so adjusting attachment data and retrying.
            – referer: http://www.xxx.com/admin/doeditattachmentstorage.action | url: /admin/doattachmentmigration.action | userName: xxx | action: doattachmentmigration

            The strange things is we were able to to migrate all attachments in the same 5.2.5 cluster on another server in March. Even though the migration failed a couple of times at the beginning, these attachments were eventually moved to DB. But this time this issue is preventing me from moving forward. Isn't the problem supposed to be resolved by 4.1 already?

            Qian Zhao added a comment - I am having the same problem when migrating attachment from file system to database on our 5.2.5 cluster. 2014-07-16 09:40:37,769 ERROR [Long running task: Attachment data migration] [confluence.util.longrunning.AttachmentMigrationLongRunningTask] runInternal Could not create Oracle LOB; nested exception is java.io.IOException: ORA-01652: unable to extend temp segment by 128 in tablespace TEMP – referer: http://www.xxx.com/admin/doeditattachmentstorage.action | url: /admin/doattachmentmigration.action | userName: xxx | action: doattachmentmigration org.springframework.dao.DataAccessResourceFailureException: Could not create Oracle LOB; nested exception is java.io.IOException: ORA-01652: unable to extend temp segment by 128 in tablespace TEMP Before the above error, I also got some warnings like this: 2014-07-16 09:38:29,376 WARN [Long running task: Attachment data migration] [persistence.dao.hibernate.AbstractHibernateAttachmentDao$IntraHibernateAttachmentCopier] saveAttachmentData Attachment size of Attachment: RAM Upgrade Process v.3 (3473418) yge is greater than 0, so adjusting attachment data and retrying. – referer: http://www.xxx.com/admin/doeditattachmentstorage.action | url: /admin/doattachmentmigration.action | userName: xxx | action: doattachmentmigration The strange things is we were able to to migrate all attachments in the same 5.2.5 cluster on another server in March. Even though the migration failed a couple of times at the beginning, these attachments were eventually moved to DB. But this time this issue is preventing me from moving forward. Isn't the problem supposed to be resolved by 4.1 already?

            CONF-9335 removes the requirement to migrate to the database for clustered Confluence.

            For migrating out of the database we will be implementing a better mechanism to help existing clustered customers. This will be delivered with the next clustered release of Confluence.

            Paul Curren added a comment - CONF-9335 removes the requirement to migrate to the database for clustered Confluence. For migrating out of the database we will be implementing a better mechanism to help existing clustered customers. This will be delivered with the next clustered release of Confluence.

            We have an open ticket: See: CSP-96637

            My vote is the same as that of David Foust - Atlassian doesn't intend on fixing this unless it is purely by accident.

            Stephen Gramm added a comment - We have an open ticket: See: CSP-96637 My vote is the same as that of David Foust - Atlassian doesn't intend on fixing this unless it is purely by accident.

            Stephen, you would be best to raise a support ticket in support.atlassian.com; our Confluence team will do their best to help you from there.

            James Fleming [Administrative Account] added a comment - Stephen, you would be best to raise a support ticket in support.atlassian.com ; our Confluence team will do their best to help you from there.

            Stephen Gramm added a comment - - edited

            We continue to have problems with attachment migration task. Our attempts to migrate ~650gb of attachments into the Oracle database has been running for ~55 hours.

            At this point it looks like the migration of attachments to the file system will run through out the week end.

            Is it possible to escalate this ticket? See: CSP-96637

            Stephen Gramm added a comment - - edited We continue to have problems with attachment migration task. Our attempts to migrate ~650gb of attachments into the Oracle database has been running for ~55 hours. At this point it looks like the migration of attachments to the file system will run through out the week end. Is it possible to escalate this ticket? See: CSP-96637

            Thanks Don,

            I'll test the new version (when it's out)

            Cheers,
            Leon

            Leon Kolchinsky added a comment - Thanks Don, I'll test the new version (when it's out) Cheers, Leon

            Don Willis added a comment -

            Hi Leon and David,

            I spent a little time during 4.1 development on improving the attachment migration situation, fixing the issues that affected our own real data. As it turned out, the specific issue mentioned here, that all the attachments are migrated in one transaction, was not the major problem I encountered, and I regret to say I didn't address it. It's a real issue that will affect customers such as Leon, and thus I have left it open.

            However, I did fix several related issues that brought the migration time for our own 40GB of attachments down to a couple of hours. That's from filesystem to database or database to filesystem.

            The fixes for CONF-15937 and CONF-23783 should, I expect, enable nearly everybody to migrate their attachments in reasonable time, ~2 hours. They are in 4.1 which will be released shortly.

            For some DBMSes this configuration may be required to handle such a large transaction. No configuration was required for our Postgres instance for 40GB of attachment data.

            Looking at this issue and what both of you (Leon & David) have said about your instances it really does sound like the tempspace issue is going to continue to hurt you., but not being an Oracle DBMS I'm not positive. I suggest try it out with 4.1 and if it fails contact support.

            I think we should at least be able to get you a patch to split the file copying into multiple transactions while you do the migration, if not fix this issue properly at last.

            Sorry to have no happier news,
            Don

            Don Willis added a comment - Hi Leon and David, I spent a little time during 4.1 development on improving the attachment migration situation, fixing the issues that affected our own real data. As it turned out, the specific issue mentioned here, that all the attachments are migrated in one transaction, was not the major problem I encountered, and I regret to say I didn't address it. It's a real issue that will affect customers such as Leon, and thus I have left it open. However, I did fix several related issues that brought the migration time for our own 40GB of attachments down to a couple of hours. That's from filesystem to database or database to filesystem. The fixes for CONF-15937 and CONF-23783 should, I expect, enable nearly everybody to migrate their attachments in reasonable time, ~2 hours. They are in 4.1 which will be released shortly. For some DBMSes this configuration may be required to handle such a large transaction. No configuration was required for our Postgres instance for 40GB of attachment data. Looking at this issue and what both of you (Leon & David) have said about your instances it really does sound like the tempspace issue is going to continue to hurt you., but not being an Oracle DBMS I'm not positive. I suggest try it out with 4.1 and if it fails contact support. I think we should at least be able to get you a patch to split the file copying into multiple transactions while you do the migration, if not fix this issue properly at last. Sorry to have no happier news, Don

              Unassigned Unassigned
              67cfe1b6089d David Foust
              Affected customers:
              16 This affects my team
              Watchers:
              15 Start watching this issue

                Created:
                Updated:
                Resolved: