Projects may be reverted to the default issue type scheme during the change due to race condition

Summary

Editing multiple different projects at the same time and assigning same IssueTypeSchema (ITS) to them, may cause data loss for other projects using same ITS. Those other projects (not the one that were changed) could be reverted to the default issue type scheme during the change due to race condition.

Steps to Reproduce

1. Generate 2000 projects.
2. Create a new issue type scheme, call it 'ITS A'.
3. Associate 1800 projects with the newly created issue type scheme 'ITS A'.
4. Jot down the number of total projects associated with the default issue type scheme (it should be 200).
5. Update two of the projects associated with the default issue type scheme before with the newly created issue type scheme 'ITS A' simultaneously via Project settings > Issue types > Actions > Use a different scheme.
   • You might need another user to do this with you at the same time.
   • Example screenshot
     
6. Review the number of total projects associated with the default issue type scheme.

Expected Results

The number of total projects associated with the default issue type scheme is reduced by 2.

Actual Results

The number of total projects associated with the default issue type scheme increases by random number (more then 200).
Enabling DEBUG logging for com.atlassian.jira.issue.context.persistence will show number of deleted rows, snippet from atlassian-jira.log file:

2018-01-11 17:24:51,050 http-nio-8081-exec-13 DEBUG admin 1044x992x1 1mxuaim 127.0.0.1 /secure/admin/SelectIssueTypeSchemeForProject.jspa [c.a.j.i.context.persistence.FieldConfigContextPersisterWorker] 1 contexts deleted for field config scheme with id '11500'
2018-01-11 17:26:41,157 http-nio-8081-exec-13 DEBUG admin 1046x1017x1 1mxuaim 127.0.0.1 /secure/admin/SelectIssueTypeSchemeForProject.jspa [c.a.j.i.context.persistence.FieldConfigContextPersisterWorker] 0 contexts deleted for field config scheme with id '11700'

Notes

• There is a race-condition between deleting existing IssueTypeSchema for other projects and reading projects association.
  • Pseudo-code:
    • Thread1 reads valid project list, deletes it and re-creates it

      // reads valid project list
         associatedProjects = configScheme.getAssociatedProjectObjects();
      ...
      // deletes all projects with schema 
         contextPersister.removeContextsForConfigScheme(newScheme); 
      <-- Thread2 reads the partial project list -->
      // re-creates projects 
         storeAssociatedContexts(configScheme, associatedProjects, field);
      

    • Thread2 reads partial project list, deletes it and re-creates partial project list

         associatedProjects = configScheme.getAssociatedProjectObjects();
      ...
      

• Problem is easier to reproduce in Datacenter due to distributed scope and delay in cache propagation.

Workaround

• Since problems is a race condition, you can mitigate it by editing IssueTypeSchema for only one projects at the same time
  • In worse case scenario to prevent the schema change, you may want to block URL /secure/admin/SelectIssueTypeSchemeForProject.jspa

Action for data recovery

• In simple scenario you can reassign corresponding IssueTypeSchema (ITS) to affected projects
• In case of large number projects affected, usually the list it's not that obvious. You will need JIRA DB backup (or staging copy).
  • You will need a couple of tables (configurationcontext, project, fieldconfigscheme) and run the following SQL to get the mapping list:

    select p.id as PID,p.pname as ProjectName,cc.fieldconfigscheme as ITS_ID,fcs.configname as ITS_Name from configurationcontext cc,project p,fieldconfigscheme fcs where cc.customfield = 'issuetype' and p.id=cc.project and fcs.id=cc.fieldconfigscheme;
      pid  |    projectname     | its_id |                   its_name
    -------+--------------------+--------+----------------------------------------------
     10001 | Dragons            |  10101 | DRA: Simple Issue Tracking Issue Type Scheme
     10100 | PRG                |  10300 | PRG: Simple Issue Tracking Issue Type Scheme
     10300 | Agile-Project      |  10805 | AG: Scrum Issue Type Scheme