Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-71998

If a failure is sustained while creating a project from shared configuration, Issue Type Schemes may be reverted to Default

XMLWordPrintable

    • 7.06
    • 18
    • Severity 1 - Critical
    • 26
    • Hide
      Atlassian Update – 15 December 2023

      Dear Customers,
      We're happy to announce that this issue is fixed in 9.14.0 release. Learn more about the release

      Best regards
      Jakub Sildatk
      Jira DC Software Engineer

      Show
      Atlassian Update – 15 December 2023 Dear Customers, We're happy to announce that this issue is fixed in 9.14.0 release. Learn more about the release Best regards Jakub Sildatk Jira DC Software Engineer

      Issue Summary

      If a failure occurs during a particular part of the Create project based on shared configuration process, other projects associated with the Issue Type Scheme of the project may be reverted to "Default Issue Type Scheme"

      The failure occurs because from a SQL perspective, Jira DELETE's existing Issue Type Scheme associations before INSERTing the updated list again, in batches. The associations are stored in the configurationcontext table.

      Failures that could trigger the problem include database problem,(for instance, an INSERT query failing due to UNDO table space error, on Oracle) or network outage between Jira and the database.

      This DELETE-then-INSERT process is not protected by a transaction, which is why the scheme associations can be reverted in this way.

      Steps to Reproduce

      1. Create 1000 projects, all on the one Issue Type Configuration Scheme
      2. Select Projects -> Create Project -> Create based on shared configuration
      3. Select one of the 1000 projects, and start the creation process.
      4. While the creation is in progress, simulate a failure to the database during the store() method of FieldConfigContextPersisterWorker.java during storeAllContexts()

      Expected Results

      Jira should fail gracefully - the Issue Type Scheme for the project selected in Create from shared configuration is not changed.

      Actual Results

      The Issue Type Scheme for the project selected in Create from shared configuration from no longer lists all projects. Some previously assigned projects are reverted back to "Default Issue Type Scheme"

      The createshared REST endpoint used to process the project creation may throw an error with associated stack trace, for example:

      2020-05-07 14:42:25,692 http-nio-8080-exec-600 url:/rest/project-t...createshared/17370 username:admin ERROR admin 881x8509894x120 11pz9go 0.0.0.0,0.0.0.0,127.0.0.1 /rest/project-templates/1.0/createshared/17370 [c.a.p.r.c.error.jersey.ThrowableExceptionMapper] Uncaught exception thrown by REST service: org.ofbiz.core.entity.GenericEntityException: while inserting: [GenericEntity:ConfigurationContext][fieldconfigscheme,13400][project,25117][id,3705875][key,issuetype] (SQL Exception while executing the following:INSERT INTO configurationcontext (ID, PROJECTCATEGORY, PROJECT, customfield, FIELDCONFIGSCHEME) VALUES (?, ?, ?, ?, ?) (ORA-30036: unable to extend segment by 128 in undo tablespace 'UNDO_3'
    ))
com.atlassian.jira.exception.DataAccessException: org.ofbiz.core.entity.GenericEntityException: while inserting: [GenericEntity:ConfigurationContext][fieldconfigscheme,13400][project,25117][id,3705875][key,issuetype] (SQL Exception while executing the following:INSERT INTO configurationcontext (ID, PROJECTCATEGORY, PROJECT, customfield, FIELDCONFIGSCHEME) VALUES (?, ?, ?, ?, ?) (ORA-30036: unable to extend segment by 128 in undo tablespace 'UNDO_3'
))
	at com.atlassian.jira.ofbiz.DefaultOfBizDelegator.createValue(DefaultOfBizDelegator.java:299)
	at com.atlassian.jira.ofbiz.WrappingOfBizDelegator.createValue(WrappingOfBizDelegator.java:172)
	at com.atlassian.jira.issue.context.persistence.FieldConfigContextPersisterWorker.store(FieldConfigContextPersisterWorker.java:105)
	at com.atlassian.jira.issue.context.persistence.CachingFieldConfigContextPersister.store(CachingFieldConfigContextPersister.java:180)
	at com.atlassian.jira.issue.fields.config.manager.FieldConfigSchemeManagerImpl.storeAssociatedContexts(FieldConfigSchemeManagerImpl.java:253)
	at com.atlassian.jira.issue.fields.config.manager.FieldConfigSchemeManagerImpl.updateFieldConfigScheme(FieldConfigSchemeManagerImpl.java:156)
	at com.atlassian.jira.bc.project.ProjectSchemeAssociationManager.lambda$associateIssueTypeScheme$7(ProjectSchemeAssociationManager.java:95)
	at java.util.Optional.ifPresent(Optional.java:159)
	at com.atlassian.jira.bc.project.ProjectSchemeAssociationManager.associateIssueTypeScheme(ProjectSchemeAssociationManager.java:90)
	at com.atlassian.jira.bc.project.ProjectSchemeAssociationManager.associateSchemesOfExistingProjectWithNewProject(ProjectSchemeAssociationManager.java:73)
	at com.atlassian.jira.bc.project.DefaultProjectService.lambda$createProject$2(DefaultProjectService.java:344)
	at com.atlassian.jira.bc.project.DefaultProjectService.createProjectInternal(DefaultProjectService.java:374)
	at com.atlassian.jira.bc.project.DefaultProjectService.createProject(DefaultProjectService.java:349)

      Replication Notes
      If SQL Debug mode is enabled, we can observe how Jira DELETEs existing associations for the scheme first, then INSERT the associations back.
      atlassian-jira-sql.log

      2020-05-19 17:29:57,213+1000 https-jsse-nio-8853-exec-2 admin 1049x769x1 180smj /rest/project-templates/1.0/createshared/10901 1ms "DELETE FROM public.configurationcontext WHERE (customfield =  'issuetype' ) AND (PROJECT IN ('10900', '10901', '10902') )"
2020-05-19 17:29:57,214+1000 https-jsse-nio-8853-exec-2 admin 1049x769x1 180smj /rest/project-templates/1.0/createshared/10901 call stack
        ...
        at com.atlassian.jira.issue.context.persistence.FieldConfigContextPersisterWorker.removeContextsForField(FieldConfigContextPersisterWorker.java:184)
        at com.atlassian.jira.issue.context.persistence.FieldConfigContextPersisterWorker.lambda$store$0(FieldConfigContextPersisterWorker.java:110)
        at java.util.Iterator.forEachRemaining(Iterator.java:116)
        at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
        at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
        at com.atlassian.jira.issue.context.persistence.FieldConfigContextPersisterWorker.store(FieldConfigContextPersisterWorker.java:109)
        at com.atlassian.jira.issue.context.persistence.CachingFieldConfigContextPersister.store(CachingFieldConfigContextPersister.java:181)
        at com.atlassian.jira.issue.fields.config.manager.FieldConfigSchemeManagerImpl.storeAssociatedContexts(FieldConfigSchemeManagerImpl.java:253)
        at com.atlassian.jira.issue.fields.config.manager.FieldConfigSchemeManagerImpl.updateFieldConfigScheme(FieldConfigSchemeManagerImpl.java:156)
        at com.atlassian.jira.bc.project.ProjectSchemeAssociationManager.lambda$associateIssueTypeScheme$7(ProjectSchemeAssociationManager.java:95)
        at java.util.Optional.ifPresent(Optional.java:159)
        at com.atlassian.jira.bc.project.ProjectSchemeAssociationManager.associateIssueTypeScheme(ProjectSchemeAssociationManager.java:90)
        at com.atlassian.jira.bc.project.ProjectSchemeAssociationManager.associateSchemesOfExistingProjectWithNewProject(ProjectSchemeAssociationManager.java:73)
        

2020-05-19 17:29:57,225+1000 https-jsse-nio-8853-exec-2 admin 1049x769x1 180smj /rest/project-templates/1.0/createshared/10901 1ms "INSERT INTO public.configurationcontext (ID, PROJECTCATEGORY, PROJECT, customfield, FIELDCONFIGSCHEME) VALUES ('11705', 'null', '10902', 'issuetype', '11700')"
2020-05-19 17:29:57,226+1000 https-jsse-nio-8853-exec-2 admin 1049x769x1 180smj /rest/project-templates/1.0/createshared/10901 call stack
    	...
    	at com.atlassian.jira.issue.context.persistence.FieldConfigContextPersisterWorker.storeAllContexts(FieldConfigContextPersisterWorker.java:197)
    	at com.atlassian.jira.issue.context.persistence.FieldConfigContextPersisterWorker.lambda$store$0(FieldConfigContextPersisterWorker.java:111)
    	at java.util.Iterator.forEachRemaining(Iterator.java:116)
    	at java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
    	at java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:580)
    	at com.atlassian.jira.issue.context.persistence.FieldConfigContextPersisterWorker.store(FieldConfigContextPersisterWorker.java:109)
    	at com.atlassian.jira.issue.context.persistence.CachingFieldConfigContextPersister.store(CachingFieldConfigContextPersister.java:181)
    	at com.atlassian.jira.issue.fields.config.manager.FieldConfigSchemeManagerImpl.storeAssociatedContexts(FieldConfigSchemeManagerImpl.java:253)
    	at com.atlassian.jira.issue.fields.config.manager.FieldConfigSchemeManagerImpl.updateFieldConfigScheme(FieldConfigSchemeManagerImpl.java:156)
    	at com.atlassian.jira.bc.project.ProjectSchemeAssociationManager.lambda$associateIssueTypeScheme$7(ProjectSchemeAssociationManager.java:95)
    	at java.util.Optional.ifPresent(Optional.java:159)
    	at com.atlassian.jira.bc.project.ProjectSchemeAssociationManager.associateIssueTypeScheme(ProjectSchemeAssociationManager.java:90)
    	at com.atlassian.jira.bc.project.ProjectSchemeAssociationManager.associateSchemesOfExistingProjectWithNewProject(ProjectSchemeAssociationManager.java:73)

      Notes

      See JRASERVER-71178 for description of the underlying problem

      Workaround

      Currently there is no known workaround to prevent this behavior. A workaround will be added here when available.

      However, the scheme associations reverted can be set back.

      Recovering the Issue Type Scheme associations

      • In simple scenario you can reassign corresponding IssueTypeSchema (ITS) to affected projects
      • In case of large number projects affected, usually the list it's not that obvious.
        • If a stack trace of the failure is found, it may include the Issue Type Scheme ID. The following example tells us the issuetype scheme ID is 13400, which means that we need to re-associate projects to the Issue Type Scheme with id 13400 
          org.ofbiz.core.entity.GenericEntityException: while inserting: [GenericEntity:ConfigurationContext][fieldconfigscheme,13400][project,25117][id,3705875][key,issuetype]
        • You can use a recent JIRA DB backup or staging copy to obtain the list of correct assignments. You will need a couple of tables (configurationcontext, project, fieldconfigscheme) and run the following SQL to get the mapping list: 
          select p.id as PID,p.pname as ProjectName,cc.fieldconfigscheme as ITS_ID,fcs.configname as ITS_Name from configurationcontext cc,project p,fieldconfigscheme fcs where cc.customfield = 'issuetype' and p.id=cc.project and fcs.id=cc.fieldconfigscheme;
  pid  |    projectname     | its_id |                   its_name
-------+--------------------+--------+----------------------------------------------
 10001 | Dragons            |  10101 | DRA: Simple Issue Tracking Issue Type Scheme
 10100 | PRG                |  10300 | PRG: Simple Issue Tracking Issue Type Scheme
 10300 | Agile-Project      |  10805 | AG: Scrum Issue Type Scheme
      • Once the affected Issue Type Scheme, and a list of projects to be fixed has been determined, set them back to the correct Issue Type Scheme via Jira Admin > Issues > Issue Type Schemes, or programmatically via the REST API: PUT /rest/api/2/issuetypescheme/(schemeId)/associations
      • If users have created new issues that have issue types that do not exist in the issue type scheme you are trying to associate in the REST API, you will need to navigate to manually update the issue type scheme for these projects manually in the project settings.

            [JRASERVER-71998] If a failure is sustained while creating a project from shared configuration, Issue Type Schemes may be reverted to Default

            Grzegorz Kuliński (Inactive) added a comment -
            Atlassian Update – 30 May 2023

            Dear Customers,

            Thank you for taking the time to file and comment on this issue. We realize it still occurs and impacts your organization. We are now working on multiple customer requests and on new features, so we have to postpone our resolution of this issue. We’ve decided to move this issue to our long-term backlog.

            Please continue watching this ticket for future updates and changes in the timeline that impacts your work.

            Best regards
            Grzegorz Kulinski
            Jira DC developer

            Grzegorz Kuliński (Inactive) added a comment - Atlassian Update – 30 May 2023 Dear Customers, Thank you for taking the time to file and comment on this issue. We realize it still occurs and impacts your organization. We are now working on multiple customer requests and on new features, so we have to postpone our resolution of this issue. We’ve decided to move this issue to our long-term backlog. Please continue watching this ticket for future updates and changes in the timeline that impacts your work. Best regards Grzegorz Kulinski Jira DC developer

            Matthias Gerlach added a comment -

            We are operating a Jira Datacenter installation with 10000 projects. Our main issue type scheme is asscociated with over 6000 projects.

            We stumbled upon this issue via the related ticket https://jira.atlassian.com/browse/JRASERVER-71178. As we had a couple of projects with no associated Issue Type Scheme.

            When assigning the main issue type scheme to those projects we noticed several things:

            • All 6000 projects were temporarily set back to the Default issue type scheme
            • The http request triggering the change timed out
            • It takes several minutes until all projects are back on the main issue type scheme.

            When looking at the SQL queries we saw the rather destructive DELETE command. And the following inserts.

            This behaviour poses a rather high threat on our platform as:

            • We are using Autoscaling(A termination of the node would trigger this issue)
            • We offer a self service to create projects with the main issue type scheme (We do not have control when this happens)
            • The whole operation takes rather long when the platform is under load. Probably because of the required database connections. (On our test environment the operation just take a couple of seconds)

            All in all this is a rather big issue for our platform and it should be prioritized.

            Matthias Gerlach added a comment - We are operating a Jira Datacenter installation with 10000 projects. Our main issue type scheme is asscociated with over 6000 projects. We stumbled upon this issue via the related ticket https://jira.atlassian.com/browse/JRASERVER-71178 . As we had a couple of projects with no associated Issue Type Scheme. When assigning the main issue type scheme to those projects we noticed several things: All 6000 projects were temporarily set back to the Default issue type scheme The http request triggering the change timed out It takes several minutes until all projects are back on the main issue type scheme. When looking at the SQL queries we saw the rather destructive DELETE command. And the following inserts. This behaviour poses a rather high threat on our platform as: We are using Autoscaling(A termination of the node would trigger this issue) We offer a self service to create projects with the main issue type scheme (We do not have control when this happens) The whole operation takes rather long when the platform is under load. Probably because of the required database connections. (On our test environment the operation just take a couple of seconds) All in all this is a rather big issue for our platform and it should be prioritized.

              9e3a0496a19c Jakub Sildatk
              allewellyn@atlassian.com Alex [Atlassian,PSE]
              Affected customers:
              15 This affects my team
              Watchers:
              30 Start watching this issue

                Created:
                Updated:
                Resolved: