Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-70807

We need a more robust process for refreshing prod data to lower environments

    • 8
    • We collect Jira feedback from various sources, and we evaluate what we've collected when planning our product roadmap. To understand how this piece of feedback will be reviewed, see our Implementation of New Features Policy.

      We recommend creating a test and dev instances of Jira prod data. This helps our Enterprise customers test upgrades and features, workarounds and fixes, I'd say "have you vetted this in a lower environment?" is one of the most common phrases I use.  The problem arises is we only take safety precautions in regard to the mail, we can set a flag in the JVM

      -Datlassian.mail.senddisabled
      -Datlassian.mail.fetchdisabled
      -Datlassian.mail.popdisabled 

      or we can disable it upon XML restore. But this still leaves the mail server and handler data in the DB and any admin can easily re-enable these mail servers. 

      There are other concerns when refreshing data:

      • DVCS syncs
      • applinks 
      • cache replication issues if prod and dev can communicate over 40001 and 40011
      • index replication if you're doing a DB dump Jira has several bugs where it will continue to try to communicate to "offline or inactive" nodes even though they cannot reach the DB see   JRASERVER-42916 - Stale node ids should automatically be removed in Jira Data Center IN PROGRESS  for more information, this can even lead to dev indexing showing up in the admin/index section in other environments 

      Suggested Action: 

      Perhaps we could create a "lower environment" exporter or importer that strips the following from the export or import or a startup/setup task that can be run manually.

      • DVCS data
      • applinks 
      • cluster node tables 
      • mail server info and mail handlers

      Workaround for Manual DB refreshes: 

      If you're not using JIRA's XML you can do the following, if you're using an XML restore, you can perform the following after importing the data and stopping Jira.

      1. Refresh the data from prod to lower environment
      2. run the following before starting Jira

      Index related tables for Data Center:

      Truncate the following tables:

      clusternode
      clusternodeheartbeat
      replicatedindexoperation
      

      Mail server info

      1. Delete all incoming and outgoing mail servers by truncating the following table:

      mailserver

      2. Remove mail handlers

      delete from serviceconfig where clazz='com.atlassian.jira.service.services.mail.MailFetcherService'

      Remove Applinks

      1. Find the applink ids:

      SELECT SUBSTR(a.property_key,16,36) as "Application Key", b.propertyvalue as "Application Name" FROM propertyentry a join propertystring b on a.id=b.id where a.property_key like 'applinks.admin%name';

      2. After identifying the application key of the application that requires removing using the above SQL in the diagnosis section, run the following queries exactly in this order, you'll also need to run these for each result from query 1.

      DELETE FROM oauthspconsumer WHERE consumer_key IN (SELECT propertyvalue FROM propertystring WHERE id IN (SELECT id FROM propertyentry WHERE property_key LIKE 'applinks.%resultsfromfirstquery%'));
      -- NOTE: For Microsoft SQLServer use the following query:
      DELETE FROM oauthspconsumer WHERE consumer_key IN (SELECT CONVERT(NVARCHAR(MAX), propertyvalue) FROM propertystring WHERE id IN (SELECT id FROM propertyentry WHERE property_key LIKE 'applinks.%resultsfromfirstquery%'));
      DELETE FROM oauthconsumertoken WHERE token_key LIKE '%resultsfromfirstquery%';
      DELETE FROM oauthconsumertoken WHERE token_key LIKE '%resultsfromfirstquery%';
      DELETE FROM trustedapp WHERE application_id IN (SELECT propertyvalue FROM propertystring WHERE id IN (SELECT id FROM propertyentry WHERE property_key LIKE 'applinks.%resultsfromfirstquery%')); 
      -- NOTE: For Microsoft SQLServer use the following query:
      DELETE FROM trustedapp WHERE application_id IN (SELECT CONVERT(NVARCHAR(MAX), propertyvalue) FROM propertystring WHERE id IN (SELECT id FROM propertyentry WHERE property_key LIKE 'applinks.%resultsfromfirstquery%'));
      DELETE FROM propertystring WHERE id IN (SELECT id FROM propertyentry WHERE property_key LIKE 'applinks.%resultsfromfirstquery%');
      DELETE FROM propertyentry WHERE property_key LIKE 'applinks.admin.resultsfromfirstquery1%';
      UPDATE propertystring SET propertyvalue = REPLACE(propertyvalue,'\n2c66970e-35f8-365f-bc65-f535d7edf1a1','') where id in (select id from propertyentry where property_key like 'applinks.global%'); 

      Remove DVCS ACCOUNTS: 

      1. Delete the orgs from "AO_E8B6CC_ORGANIZATION_MAPPING":
        delete from "AO_E8B6CC_ORGANIZATION_MAPPING"; 
      2. Delete the repos from "AO_E8B6CC_REPOSITORY_MAPPING":
        delete from "AO_E8B6CC_REPOSITORY_MAPPING""

        This will remove all connection information from JIRA to GHE.

      AFTER these steps have been done you can start Jira for the first time. 

      This should remove all mail servers, dvcs account info and applinks from the instance, we also have a comprehensive guide for this here.

          Form Name

            [JRASERVER-70807] We need a more robust process for refreshing prod data to lower environments

            Want a simple disable/enable button on the DVCS accounts page so we can turn each account on/off as needed in lower environments. This should also be available via the API.

            Aaron Matthys added a comment - Want a simple disable/enable button on the DVCS accounts page so we can turn each account on/off as needed in lower environments. This should also be available via the API.

            Yeah I see that argument for sure, esp for CPrime's use case handling several instances vs the admin of a single cluster. Having something that works across different DBMS' is less valuable in that case.

            Alex Gallien added a comment - Yeah I see that argument for sure, esp for CPrime's use case handling several instances vs the admin of a single cluster. Having something that works across different DBMS' is less valuable in that case.

            My major point in DB vs REST is that when Atlassian or app vendors make changes to their DB structure (and they do this silently in most of the cases), it is much safer to rely on the public API than troubleshoot failed data refresh/recovery process. At least API gets documented (mostly). Plus, REST API is more versatile and doesn't require adjustments to different DB engines.

            Yevgen Lasman added a comment - My major point in DB vs REST is that when Atlassian or app vendors make changes to their DB structure (and they do this silently in most of the cases), it is much safer to rely on the public API than troubleshoot failed data refresh/recovery process. At least API gets documented (mostly). Plus, REST API is more versatile and doesn't require adjustments to different DB engines.

            Hard disagree on the mail aspect - there is no reason a lower env should ever point to an active Prod mailbox, so sanitizing that to avoid any potential of lower envs scooping up Prod mail is the way to go imo. REST could totally handle most of this, but doing this all in SQL you:

            • Can run it all in one command instead of scripting multiple REST calls.
            • Are already mostly configured upon first startup of the refreshed instance.

            Plus you already have a cold db to modify so very low risk.

            Alex Gallien added a comment - Hard disagree on the mail aspect - there is no reason a lower env should ever  point to an active Prod mailbox, so sanitizing that to avoid any potential of lower envs scooping up Prod mail is the way to go imo. REST could totally handle most of this, but doing this all in SQL you: Can run it all in one command instead of scripting multiple REST calls. Are already mostly configured upon first startup of the refreshed instance. Plus you already have a cold db to modify so very low risk.

            Most of these changes can be done through REST API, including licenses switch (prod -> dev), application links, base url and banner updates, etc. And it is better not to delete but turn off or reconfigure (depending in requirements) settings like email, SSO and others.

            Yevgen Lasman added a comment - Most of these changes can be done through REST API, including licenses switch (prod -> dev), application links, base url and banner updates, etc. And it is better not to delete but turn off or reconfigure (depending in requirements) settings like email, SSO and others.

            We use OneLogin so can't speak for Okta specifically, but afaik when it's set to secondary you just get a login screen and no SAML calls are made unless you explicitly hit the test link provided in the SAML config page. Haven't explicitly tested that though so YMMV.

            Alex Gallien added a comment - We use OneLogin so can't speak for Okta specifically, but afaik when it's set to secondary you just get a login screen and no SAML calls are made unless you explicitly hit the test link provided in the SAML config page. Haven't explicitly tested that though so YMMV.

            Matt Doar added a comment -

            True. But does it cause lots of failing connections to Okta? I'm a bit worried about okta rate limiting

            Matt Doar added a comment - True. But does it cause lots of failing connections to Okta? I'm a bit worried about okta rate limiting

            Matt - this query flips SSO to secondary auth which pretty much accomplishes the same thing, we run it whenever we refresh lower envs:

            UPDATE propertystring SET propertyvalue = 'false' WHERE ID = (SELECT ID FROM propertyentry WHERE property_key = 'com.atlassian.plugins.authentication.samlconfig.redirect-on-login');

            Alex Gallien added a comment - Matt - this query flips SSO to secondary auth which pretty much accomplishes the same thing, we run it whenever we refresh lower envs: UPDATE propertystring SET propertyvalue = ' false ' WHERE ID = (SELECT ID FROM propertyentry WHERE property_key = 'com.atlassian.plugins.authentication.samlconfig.redirect-on-login' );

            Matt Doar added a comment -

            Another case is where you have SSO enabled in production but not in staging. You need to disable SSO in the lower environment. I think it can be done by deleting these 8 rows but I'm not sure and have not tested this.

            select propertyentry.* from propertyentry join propertystring on propertyentry.id=propertystring.id where property_key like 'com.atlassian.plugins.authentication.samlconfig.%';
            

            Matt Doar added a comment - Another case is where you have SSO enabled in production but not in staging. You need to disable SSO in the lower environment. I think it can be done by deleting these 8 rows but I'm not sure and have not tested this. select propertyentry.* from propertyentry join propertystring on propertyentry.id=propertystring.id where property_key like 'com.atlassian.plugins.authentication.samlconfig.%';

            Hey stephen.bell1, yeah not sure how that got missed, you can just delete the two tables mentioned above:

            1. AO_E8B6CC_ORGANIZATION_MAPPING
            2. AO_E8B6CC_REPOSITORY_MAPPING
              before starting JIRA.

            Ronnie Volkmar added a comment - Hey stephen.bell1 , yeah not sure how that got missed, you can just delete the two tables mentioned above: AO_E8B6CC_ORGANIZATION_MAPPING AO_E8B6CC_REPOSITORY_MAPPING before starting JIRA.

              Unassigned Unassigned
              rvolkmar Ronnie Volkmar
              Votes:
              31 Vote for this issue
              Watchers:
              33 Start watching this issue

                Created:
                Updated: