• 16
    • 7
    • We collect Confluence feedback from various sources, and we evaluate what we've collected when planning our product roadmap. To understand how this piece of feedback will be reviewed, see our Implementation of New Features Policy.

      Suggestion Description

      It has been painful for customers with large dataset to migrate from an external database to another, for example, migrate from MySQL to PostgreSQL. The current process consists of taking an XML backup and importing it to an empty instance pointing to the new database and then, after import is done, point the existing instance to the new database. Problem here is that XML Backups for large instances may take days and during these days, new data is being written to the instance, since huge instances can't be turned into read-only instances. It would be a major win for Atlassian customers to have a Migration Wizard Tool to transport data from one database to another through Confluence UI.

      Suggestion Details

      BitBucket is an application that has this implementation and that allows administrators to easily migrate from one external database to another. Process is documented in the Using the Database Migration Wizard BitBucket documentation. As this seems to be an implementation unrelated to the application itself, it may be possible to implement the same solution in other Atlassian applications, like Confluence.

      Suggestion Importance

      Customers with huge datasets usually end up by developing their own solutions when upon the need of migrating to a different database, which turns into a project at their side that has costs and may turn into frustration using our tools. It would be a major win for everyone using Confluence to have this feature.

      BitBucket Wizard Example

            [CONFSERVER-54249] Database Migration Wizard for Confluence

            Dan C added a comment -

            I'm looking for this concept for Jira too... currently trying to use pgloader, but based on these comments, maybe I should stop.

            Dan C added a comment - I'm looking for this concept for Jira too... currently trying to use pgloader, but based on these comments, maybe I should stop.

            Our issue is our instances are so big that the XML backup tools don't even work. It's not a matter of things taking days, it's that they simply don't work at scale. We will likely have solved the issue of migration between DBs by the time any Atlassian solution gets developed but the pain and that this involves makes us believe that it should be a built in feature.

            Matthew Thompson added a comment - Our issue is our instances are so big that the XML backup tools don't even work. It's not a matter of things taking days, it's that they simply don't work at scale. We will likely have solved the issue of migration between DBs by the time any Atlassian solution gets developed but the pain and that this involves makes us believe that it should be a built in feature.

            Eric Lemay added a comment -

            Update here after some more work has been done trying to get it to migrate, I'd echo the sentiment expressed by @Jason Smith.  pgloader has proven to be extremely finicky and has a slew of issues that seemingly crop up at random and simply make the conversion fail, making it a solution that would be difficult to recommend for a production deployment.

            Eric Lemay added a comment - Update here after some more work has been done trying to get it to migrate, I'd echo the sentiment expressed by @Jason Smith.  pgloader has proven to be extremely finicky and has a slew of issues that seemingly crop up at random and simply make the conversion fail, making it a solution that would be difficult to recommend for a production deployment.

            We have similarly wasted tens of hours with pgloader - there always seems to be some problem at the end of it.

            Jason D Smith added a comment - We have similarly wasted tens of hours with pgloader - there always seems to be some problem at the end of it.

            Eric Lemay added a comment -

            +1 Here.  The fact that this functionality exists for Bitbucket implies that it should be possible to roll out across the suite.  It would take some work though.  Working on doing this with PGLoader with limited success , there are some tables that are clearly not migrating correctly and debugging it is very long and tedious.  Given the near impossibility of generating a full site backup when the site is large, and the fact that there is no other way to migrate global settings, this is something that is definitely needed.

            Eric Lemay added a comment - +1 Here.  The fact that this functionality exists for Bitbucket implies that it should be possible to roll out across the suite.  It would take some work though.  Working on doing this with PGLoader with limited success , there are some tables that are clearly not migrating correctly and debugging it is very long and tedious.  Given the near impossibility of generating a full site backup when the site is large, and the fact that there is no other way to migrate global settings, this is something that is definitely needed.

            @Hendrik:

            the problems you describe are only, if you doing space exports and imports. If you do complete xml backups and restore, these problems don't exist. 

             

            But with xml migration, there are many other problems:

            • it takes very long on big instances, which will cause very long downtimes
            • on big instances, imports will run into errors, often after hours of importing
            • many different problems can occur, which requires to modify the source database or the xml file itself.

            did anyone had luck on migrating on database level with external tools like pgloader or ESF Database Migration Toolkit?

            Tom Penndorf added a comment - @Hendrik: the problems you describe are only, if you doing space exports and imports. If you do complete xml backups and restore, these problems don't exist.    But with xml migration, there are many other problems: it takes very long on big instances, which will cause very long downtimes on big instances, imports will run into errors, often after hours of importing many different problems can occur, which requires to modify the source database or the xml file itself. did anyone had luck on migrating on database level with external tools like pgloader or ESF Database Migration Toolkit?

            This feature would be awesome. Currently there is no good solution for a clean migration. With XML-backup there are the following problems:

            • The pages get a new page ID in the new instance. So many external links are broken.
            • If Confluence is linked with JIRA, the links between a page and an issue are broken and must be corrected (manually or via database).
            • Data from AddOns can not be migrated at all.

            Deleted Account (Inactive) added a comment - This feature would be awesome. Currently there is no good solution for a clean migration. With XML-backup there are the following problems: The pages get a new page ID in the new instance. So many external links are broken. If Confluence is linked with JIRA, the links between a page and an issue are broken and must be corrected (manually or via database). Data from AddOns can not be migrated at all.

            "Customers with huge datasets usually end up by developing their own solutions when upon the need of migrating to a different database, which turns into a project at their side that has costs and may turn into frustration using our tools."

            This is precisely my current frustration.  Trying to get a Confluence instance with a 32GB BODYCONTENT table to perform a XML backup is simply not possible.  Using tools like pgloader for MySQL -> PostgreSQL still requires DBA knowledge as it is not a clean conversion without a lot of tweaking (which I still have not gotten through).

            Bitbucket and Fisheye/Crucible have had this option for a while... would really be nice to see some unity across the apps and have this in Confluence and JIRA.

            Jesse Rehmer [Contegix] added a comment - "Customers with huge datasets usually end up by developing their own solutions when upon the need of migrating to a different database, which turns into a project at their side that has costs and may turn into frustration using our tools." This is precisely my current frustration.  Trying to get a Confluence instance with a 32GB BODYCONTENT table to perform a XML backup is simply not possible.  Using tools like pgloader for MySQL -> PostgreSQL still requires DBA knowledge as it is not a clean conversion without a lot of tweaking (which I still have not gotten through). Bitbucket and Fisheye/Crucible have had this option for a while... would really be nice to see some unity across the apps and have this in Confluence and JIRA.

              Unassigned Unassigned
              mhorlle Marcelo Horlle
              Votes:
              75 Vote for this issue
              Watchers:
              52 Start watching this issue

                Created:
                Updated: