Uploaded image for project: 'Confluence Data Center'
  1. Confluence Data Center
  2. CONFSERVER-28954

Keystore table is empty post migrating from Confluence Cloud

      I have had a stupendously shitty experience trying to import OnDemand data into local JIRA and Confluences. In this bug report I've documented the problems, what I think OnDemand developers could do to fix them in bold, and how I worked around them.

      The problem is that after importing a Confluence Could export into Confluence (5.0.2), you'll find all jiraissues macros broken (note: I had previously migrated JIRA data to its new home):

      and a cryptic error in the logs:

      java.lang.NullPointerException: currentApplication
      	at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:204)
      	at com.atlassian.applinks.core.auth.trusted.TrustedApplicationsRequestFactory.<init>(TrustedApplicationsRequestFactory.java:31)
      	at com.atlassian.applinks.core.auth.trusted.TrustedAppsAuthenticationProviderPluginModule$1.getRequestFactory(TrustedAppsAuthenticationProviderPluginModule.java:80)
      	at com.atlassian.applinks.core.auth.ApplicationLinkRequestFactoryFactoryImpl.getApplicationLinkRequestFactory(ApplicationLinkRequestFactoryFactoryImpl.java:53)
      

      With a bit of thinking one might figure out the error: the applinks are all broken, as is to be expected with the change of host.

      Suggested fix 1: improve the jiraissues error message when applinks have problems

      So off we go to the applinks page:

      Applinks are not editable, because they are marked 'system'.

      Suggested fix 2: The Confluence Cloud exporter should remove the 'system' flag from the applink, so mortals can edit them

      The 'system' flag can be removed manually in the database:

      mysql> select BANDANAKEY,BANDANAVALUE from BANDANA WHERE bandanakey like 'applinks.%.system';
      +------------------------------------------------------------------+-----------------------+
      | BANDANAKEY                                                       | BANDANAVALUE          |
      +------------------------------------------------------------------+-----------------------+
      | applinks.admin.30e7e368-6e59-357d-b986-d3374f95a205.system       | <string>true</string> |
      | applinks.application.30e7e368-6e59-357d-b986-d3374f95a205.system | <string>true</string> |
      +------------------------------------------------------------------+-----------------------+
      2 rows in set (0.00 sec)
      mysql> delete from BANDANA WHERE bandanakey like 'applinks.%.system';Query OK, 2 rows affected (0.01 sec)
      

      Now I can edit the applink:

      And then my problems really began.

      For a start, the applink URL isn't editable:

      (Questionable design decision here: sure, changing the URL to a different JIRA would break everything, but most times it's the same JIRA, just under a different URL.)

      So anyway, let's edit the URL in the database, first by finding the applink ID, finding the applink's properties, then changing the URLs:

      mysql> SELECT BANDANAVALUE FROM BANDANA WHERE BANDANAKEY = 'applinks.global.application.ids';
      +------------------------------------------------------------------------+
      | BANDANAVALUE                                                           |
      +------------------------------------------------------------------------+
      | <list>
        <string>30e7e368-6e59-357d-b986-d3374f95a205</string>
      </list> |
      +------------------------------------------------------------------------+
      1 row in set (0.00 sec)
      mysql> update BANDANA set BANDANAVALUE=replace(BANDANAVALUE, 'somecompany.atlassian.net', 'jira.somecompany.com') where BANDANAKEY like '%30e7e368-6e59-357d-b986-d3374f95a205%.url';
      Query OK, 2 rows affected (0.01 sec)
      Rows matched: 2  Changed: 2  Warnings: 0
      

      (and whatever you do, don't change the applink's title-even though it contains an incorrect URL too-the title is embedded in hundreds of Confluence page jiraissues macros. That is a job for bulk-rewriting.)

      After a Confluence restart, the applink appears with the correct link.

      Switching to the 'Outgoing Authentication' tab, it claims to be in Status: Configured. Just to be sure let's disable and re-enable it. Clicking 'Disable' is easy enough, but then:

      Where is the 'Enable' link??

      This is a UI bug (affects Firefox 16.0.1, Chrome 26.0.1410.63) - the link is actually there, and can be seen by clicking on the last form element (the timeout field) and pressing tab:

      Suggested fix #3: Atlassian should use a sane, modern JS library that sizes popups appropriately

      So we click 'Enable', and:

      java.lang.NullPointerException
      	at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter$CertificateServerImpl.writeCertificate(TrustedApplicationsFilter.java:201)
      	at com.atlassian.security.auth.trustedapps.filter.TrustedApplicationsFilter.doFilter(TrustedApplicationsFilter.java:90)
      	at com.atlassian.confluence.util.AbstractBootstrapHotSwappingFilter.doFilter(AbstractBootstrapHotSwappingFilter.java:30)
      

      "But hang on", I hear you say, "why are you trying to fix this obviously borked applink? Just delete it and start again". No such luck:


      (and, in case you were wondering, it breaks the same way if you don't change the URLs in the database as I did)

      "Okay, what about that 'Upgrade application link' button?" It dies with the JS equivalent of a NullPointerException:

      So we're left with an applink we can't fix and can't delete.

      Fortunately, Googling the exception brings up this KB article, which points to the cause: the #@#%! Confluence Cloud exporter has erased all the KEYSTORE records, which trustedapps depends on:

      mysql> select * from KEYSTORE;
      Empty set (0.00 sec)
      

      The KB article points to CONF-11074, which helpfully includes a JSP to recreate the KEYSTORE.

      So I run the JSP, which populates my KEYSTORE table with a public/private keypair, and now - hurrah - I can enable Outgoing Authentication. After a restart, the jiraissues macros that rely on the applink work too.

      Suggested fix #4: if you have to wipe the KEYSTORE table, replace its values with something else!

      Don't just leave them blank when doing so results in a broken, undeletable, unfixable applink. Create a new unique keypair using the code from the CONF-11074 JSP and put the keys in KEYSTORE.

      Thank you for reading.

        1. al1.png
          al1.png
          21 kB
        2. al2.png
          al2.png
          29 kB
        3. al3.png
          al3.png
          31 kB
        4. al4.png
          al4.png
          30 kB
        5. al5.png
          al5.png
          54 kB
        6. al6.png
          al6.png
          47 kB
        7. al7.png
          al7.png
          55 kB
        8. al8.png
          al8.png
          82 kB
        9. al9.png
          al9.png
          107 kB
        10. al0.png
          al0.png
          12 kB
        11. al8.png
          al8.png
          28 kB

          Form Name

            [CONFSERVER-28954] Keystore table is empty post migrating from Confluence Cloud

            If this issue is resolved, why am I hitting this same problem in Confluence 7.13.2? 

            Jeanne Howe added a comment - If this issue is resolved, why am I hitting this same problem in Confluence 7.13.2? 

            @sattesh Meran (don't seem to be able to mention you correctly), thanks for the feedback. I think I hit into this issue due to a cloud import. But it was a while ago. We imported from version 5.9.0-OD-58-008. Apparently, we just haven't used any functionality (namely external gadgets) up to now that would have required thess keystore entries. Maybe (and hopefully) this issue has been resolved in the meantime. I still think it might have been worth integrating a proper health check into confluence with a corresponding fix to recreate the entries from within Confluence. After all, the code to do so is quite minimal looking at the JSP. Would have saved me (and - judging from what I've found on the Internet - quite a few others) a couple of hours...

            Martin Meyer added a comment - @sattesh Meran (don't seem to be able to mention you correctly), thanks for the feedback. I think I hit into this issue due to a cloud import. But it was a while ago. We imported from version 5.9.0-OD-58-008 . Apparently, we just haven't used any functionality (namely external gadgets) up to now that would have required thess keystore entries. Maybe (and hopefully) this issue has been resolved in the meantime. I still think it might have been worth integrating a proper health check into confluence with a corresponding fix to recreate the entries from within Confluence. After all, the code to do so is quite minimal looking at the JSP. Would have saved me (and - judging from what I've found on the Internet - quite a few others) a couple of hours...

            Sattesh M added a comment -

            Thank you for the update, martin.meyer.

            When migrating later versions of Confluence cloud, we should no longer be hitting into this issue.
            That being said, we could be hitting into some other issues with the keystore which might be resolved by the same workaround in these issues.

            Can I know if you hit into this issue with a site import from Confluence Cloud as well, or was it some other issue?
            If you're hitting into the issue with the latest site export from Confluence Cloud, do reach out to our support team below, as we might need to look into the export itself.

            Cheers,
            Sattesh Maran

            Sattesh M added a comment - Thank you for the update, martin.meyer . When migrating later versions of Confluence cloud, we should no longer be hitting into this issue. That being said, we could be hitting into some other issues with the keystore which might be resolved by the same workaround in these issues. Can I know if you hit into this issue with a site import from Confluence Cloud as well, or was it some other issue? If you're hitting into the issue with the latest site export from Confluence Cloud, do reach out to our support team below, as we might need to look into the export itself. Contact Support Cheers, Sattesh Maran

            Martin Meyer added a comment - - edited

            Successfully restored Keystore entries with the information from this issue and the script from https://jira.atlassian.com/browse/CONFSERVER-11074. @Atlassian, how in the world is it possible that this problem isn't fixed and the solution properly integrated into Confluence by now??? The issues are from 2008 and 2013!!!

            And why the hell is this issue (along with all referenced issues) closed?

            Martin Meyer added a comment - - edited Successfully restored Keystore entries with the information from this issue and the script from  https://jira.atlassian.com/browse/CONFSERVER-11074 . @Atlassian, how in the world is it possible that this problem isn't fixed and the solution properly integrated into Confluence by now??? The issues are from 2008 and 2013!!! And why the hell is this issue (along with all referenced issues) closed?

            Hi,

            For the Fix#3 when the keystore table is empty, when you are in 5.9 OD the jsp file attached to the CONF-11074 is not working at all.
            So what to do in this case ?

            Cyrille Martin added a comment - Hi, For the Fix#3 when the keystore table is empty, when you are in 5.9 OD the jsp file attached to the CONF-11074 is not working at all. So what to do in this case ?

            MattS added a comment -

            Agreed, this is a really painful process for our customers

            MattS added a comment - Agreed, this is a really painful process for our customers

            It would be nice to get this some priority. We run into all of the mentioned issues every single time we do an import from Cloud. These items should be pretty easy for resolve.

            The issues with application links, gadget subscriptions, and the keystore table being emptied are all easily fixable and cause a lot of headaches. Migrations from Cloud become more complicated than simply importing the XML and changing the sysadmin passwords.

            Contegix Support added a comment - It would be nice to get this some priority. We run into all of the mentioned issues every single time we do an import from Cloud. These items should be pretty easy for resolve. The issues with application links, gadget subscriptions, and the keystore table being emptied are all easily fixable and cause a lot of headaches. Migrations from Cloud become more complicated than simply importing the XML and changing the sysadmin passwords.

            We've updated this BUG and linked to it CONF-11074 for better reference.

            Thanks and Regards,
            David|Confluence Support Engineer

            Deividi Luvison (Inactive) added a comment - We've updated this BUG and linked to it CONF-11074 for better reference. Thanks and Regards, David|Confluence Support Engineer

              datran Dat Tran (Inactive)
              97bf41494e76 Jeff Turner
              Affected customers:
              13 This affects my team
              Watchers:
              25 Start watching this issue

                Created:
                Updated:
                Resolved: