Uploaded image for project: 'Jira Platform Cloud'
  1. Jira Platform Cloud
  2. JRACLOUD-79847

Changing workflow that impacts large number of issues, causes issues created during workflow migration in an invalid workflow state

    • 18
    • Severity 3 - Minor
    • 33
    • Hide

      Hi everyone,

      This is Marcel from the Jira Cloud team again. I want to let you all know that the fix has been fully rolled out. However we are still monitoring one known edge case and I will hold the ticket open while we work through that.

      Show
      Hi everyone, This is Marcel from the Jira Cloud team again. I want to let you all know that the fix has been fully rolled out. However we are still monitoring one known edge case and I will hold the ticket open while we work through that.

      Issue Summary

      Changing workflow that impacts large number of issues, causes issues created during workflow migration in an invalid workflow state. These new tickets created during the migration window still points to the old workflow. Trying to transition the issue fails with the error ‘Couldn't find the workflow’

      Steps to Reproduce

      1. Go to Project Settings -> Workflows and change the workflow for any issue type.
      2. Pre-requisite: Make sure the project has more number of tickets under that specific issue type, so that the workflow migration takes a few minutes/hours.
      3. While the workflow migration happens, open a few fresh tickets in the same issue type.

      Expected Results

      The new tickets should start using the new workflow. We should be able to transition the tickets properly. 

      Actual Results

      The new tickets still points to the old workflow. We’re unable to transition the tickets, it fails with the error ‘Couldn't find the workflow’.

      Workaround 

      1. Move the ticket to a different issue type, and then revert it back to the original issue type. This would trigger workflow changes and it will fix the problem.
      2. Make a copy of the new workflow, and re-associate it to the issue type again. This would trigger the workflow migration again, and the problematic tickets will be fixed as a part of it. However, this operation might again take some time (based on the size of your project).

            [JRACLOUD-79847] Changing workflow that impacts large number of issues, causes issues created during workflow migration in an invalid workflow state

            Hey everyone,

            We have completed roll out. 

            Marcel Caroly added a comment - Hey everyone, We have completed roll out. 

            Hi everyone,

            This is Marcel from the Jira Cloud team again. I want to let you all know that the fix has been fully rolled out. However we are still monitoring one known edge case and I will hold the ticket open while we work through that.

            Marcel Caroly added a comment - Hi everyone, This is Marcel from the Jira Cloud team again. I want to let you all know that the fix has been fully rolled out. However we are still monitoring one known edge case and I will hold the ticket open while we work through that.

            Hi everyone,

            This is Marcel again from the Jira Cloud team. Thank you for your patience as we work on fixing this bug. The bug has taken longer than expected for us to fix because it ties in with a large change to the way this process works. At present we are soaking this solution with internal teams and hope to start roll-out by the 12th of April.

            I also want to be upfront as well and just say that we won't be able to completely eliminate these sorts of issues from happening, but we will have addressed some of the more common scenarios and hopefully made it more manageable for users. Our goal is to make the process of publishing a draft workflow faster, more reliable and more recoverable. Changes includes:

            • A boost in speed. We see 100K sized project updates in minutes rather than hours. Though, I don't want to oversell this as our architecture is always evolving and performance may vary in busy sites or regions.
            • A 'temporary' version of the changing workflows is made available immediately on publishing the draft. Temporary workflows maintain your workflow while we update issue state. Status mappings are converted into transitions that users can use during longer running migrations. This allows them to move issues to their desired state manually when things are taking a while or something goes wrong. We feel this is better than leaving the issues in a stuck state.
            • Increased locking around workflow changes to prevent potential clashes with workflow scheme changes.
            • Fixes that allow admins to make changes to workflow scheme that are still linked to trashed or archived projects.

            We know you have been waiting eagerly for this fix as we are, and we will be providing any updates as they come. Thank you again for your patience!
            Best,
            Marcel

            Marcel Caroly added a comment - Hi everyone, This is Marcel again from the Jira Cloud team. Thank you for your patience as we work on fixing this bug. The bug has taken longer than expected for us to fix because it ties in with a large change to the way this process works. At present we are soaking this solution with internal teams and hope to start roll-out by the 12th of April. I also want to be upfront as well and just say that we won't be able to completely eliminate these sorts of issues from happening, but we will have addressed some of the more common scenarios and hopefully made it more manageable for users. Our goal is to make the process of publishing a draft workflow faster, more reliable and more recoverable. Changes includes: A boost in speed. We see 100K sized project updates in minutes rather than hours. Though, I don't want to oversell this as our architecture is always evolving and performance may vary in busy sites or regions. A 'temporary' version of the changing workflows is made available immediately on publishing the draft. Temporary workflows maintain your workflow while we update issue state. Status mappings are converted into transitions that users can use during longer running migrations. This allows them to move issues to their desired state manually when things are taking a while or something goes wrong. We feel this is better than leaving the issues in a stuck state. Increased locking around workflow changes to prevent potential clashes with workflow scheme changes. Fixes that allow admins to make changes to workflow scheme that are still linked to trashed or archived projects. We know you have been waiting eagerly for this fix as we are, and we will be providing any updates as they come. Thank you again for your patience! Best, Marcel

            Hey everyone, I'm Marcel, an engineer from the Jira cloud team.

            I wanted to drop a bit more communication around this bug and what changes we are making to address it.

            Today, this process results in issues getting stuck because we don’t swap the workflows out of your schemes until all the issues are updated to be in statuses in the final workflow. This means, users working on issues in related projects will keep being creating and transitioning issues in the old workflows until the migration finishes. This means there is some risk that issues will be missed by the migration process and it means that failures can result in some issues getting stuck in either the old or new workflow. 

            We are bringing an approach we use in team managed projects to global workflow scheme management to resolve this. It works using a ‘temporary' copy of your final workflow that retains any workflow status that is being removed. This lets you ‘use’ what looks to be your desired workflow while we are updating issue statuses in the background. When the migration is completed they will be cleaned up by the system. 

            However, to ensure that conflicting updates can’t be run at the same time we will also block workflow edits in a variety of places during these migrations. This is something to be mindful of when performing very large updates and we will be improving messaging in a variety of places to help communicate this. 

            Marcel Caroly added a comment - Hey everyone, I'm Marcel, an engineer from the Jira cloud team. I wanted to drop a bit more communication around this bug and what changes we are making to address it. Today, this process results in issues getting stuck because we don’t swap the workflows out of your schemes until all the issues are updated to be in statuses in the final workflow. This means, users working on issues in related projects will keep being creating and transitioning issues in the old workflows until the migration finishes. This means there is some risk that issues will be missed by the migration process and it means that failures can result in some issues getting stuck in either the old or new workflow.  We are bringing an approach we use in team managed projects to global workflow scheme management to resolve this. It works using a ‘temporary' copy of your final workflow that retains any workflow status that is being removed. This lets you ‘use’ what looks to be your desired workflow while we are updating issue statuses in the background. When the migration is completed they will be cleaned up by the system.  However, to ensure that conflicting updates can’t be run at the same time we will also block workflow edits in a variety of places during these migrations. This is something to be mindful of when performing very large updates and we will be improving messaging in a variety of places to help communicate this. 

            Hey everyone, we have started working towards unblocking this bug and the error messages that propagate and impact issue creation during a migration window.  

            We will add updates as we get closer to launching.

            Roland Flanagan added a comment - Hey everyone, we have started working towards unblocking this bug and the error messages that propagate and impact issue creation during a migration window.   We will add updates as we get closer to launching.

            As an update to those following, we are continuing to work on this and will continue to update on our progress.

            Roland Flanagan added a comment - As an update to those following, we are continuing to work on this and will continue to update on our progress.

            aradu added a comment -

            We are aware of this problem and are working on a fix for this. The fix itself is quite involved so I cannot give a timeline for this being rolled out.

            TMP is not affected by this, and we will be using a similar mechanism for performing workflow updates there.

            aradu added a comment - We are aware of this problem and are working on a fix for this. The fix itself is quite involved so I cannot give a timeline for this being rolled out. TMP is not affected by this, and we will be using a similar mechanism for performing workflow updates there.

              mcaroly@atlassian.com Marcel Caroly
              66a036c522da Makesh Sundar
              Affected customers:
              4 This affects my team
              Watchers:
              21 Start watching this issue

                Created:
                Updated:
                Resolved: