• 103
    • Severity 3 - Minor
    • 11
    • Hide

      Hello everyone!

      As per my previous update, the improvements we are making to resolve this reliability issue are well underway. We'll soon be announcing an EAP program and asking for expressions for interest. I'll post the link to that here when its open. 

      if you are one of the impacted customers, below are times it may occur based on region:

      • Asia Pacific and Central Europe - Mon, Wed, Thurs, 9am AEST / 10am AEDT, 
      • Americas, Western Europe -   Mon, Wed, Thurs, 1pm AEST / 2pm AEDT

      Again, thanks very much for your continued patience with this.

      Justin King

      Senior Product Manager, Jira Service Management

      Show
      Hello everyone! As per my previous update, the improvements we are making to resolve this reliability issue are well underway. We'll soon be announcing an EAP program and asking for expressions for interest. I'll post the link to that here when its open.  if you are one of the impacted customers, below are times it may occur based on region: Asia Pacific and Central Europe - Mon, Wed, Thurs, 9am AEST / 10am AEDT,  Americas, Western Europe -   Mon, Wed, Thurs, 1pm AEST / 2pm AEDT Again, thanks very much for your continued patience with this. Justin King Senior Product Manager, Jira Service Management

      Issue Summary

      Assets Customers may be impacted by a brief period of downtime each week day when deployments occur. During this time the Assets user experience and API is not available. Objects added to Assets custom fields may fail to load and Automation rules involving Assets may encounter errors. For most customers this downtime lasts seconds. However, for some customers with larger number of objects it may take minutes. 

      Expected Results

      Customers experience no regular downtime during deployments.

      Actual Results

      There is downtime during deployments, the length of which varies based on the number and complexity of objects stored in Assets.

      Steps to Reproduce

      May occur each week day. 

      Workaround

      Currently there is no known workaround for this behaviour. A workaround will be added here when available.

            [JSDCLOUD-12246] Assets is unavailable during regular deployments

            Hi Justin,

            We are in Enterprise subscription and our users use JSM for 24/7 support, and the JSM instance is highly dependent on assets. Assets not available means no one can create tickets. I got to know from Atlassian support these activities happen during:

            • Asia Pacific and Central Europe - Mon, Wed, Thurs, 9am AEST / 10am AEDT,
            • Americas, Western Europe - Mon, Wed, Thurs, 1pm AEST / 2pm AEDT

            It would be great if Atlassian take necessary actions to these deployments causes zero downtime.

            Best Regards,

            Suraj

            Suraj Daluwakgoda added a comment - Hi Justin, We are in Enterprise subscription and our users use JSM for 24/7 support, and the JSM instance is highly dependent on assets. Assets not available means no one can create tickets. I got to know from Atlassian support these activities happen during: Asia Pacific and Central Europe - Mon, Wed, Thurs, 9am AEST / 10am AEDT, Americas, Western Europe - Mon, Wed, Thurs, 1pm AEST / 2pm AEDT It would be great if Atlassian take necessary actions to these deployments causes zero downtime. Best Regards, Suraj

            Is this related to or the cause of this issue linked below, which occurs daily from 6:45pm-10:45pm UTC?  I'd like to identify which one to follow as the parent issue and bug resolution.  Thanks.

            JSDCLOUD-14308 JSM Assets API latency causing failures in dependent operations - Create and track feature requests for Atlassian products.

            Grant Smith added a comment - Is this related to or the cause of this issue linked below, which occurs daily from 6:45pm-10:45pm UTC?  I'd like to identify which one to follow as the parent issue and bug resolution.  Thanks. JSDCLOUD-14308 JSM Assets API latency causing failures in dependent operations - Create and track feature requests for Atlassian products.

            7260236ab89d  - since there is no workaround - can we please be notified when assets will have a brief downtime AHEAD Of time?  

            This has an extremely negative impact for our customers .

             

            Allison Carlson added a comment - 7260236ab89d   - since there is no workaround - can we please be notified when assets will have a brief downtime AHEAD Of time?   This has an extremely negative impact for our customers .  

            It would be helpful to receive advance notice of such outages or to see an update on the status page indicating outages. Will Atlassian provide this information in the future?

            Max Fialkow-Temp added a comment - It would be helpful to receive advance notice of such outages or to see an update on the status page indicating outages. Will Atlassian provide this information in the future?

            Moving the deployments "out of business hours where possible" does not help us at all because we have orchestration jobs that run every hour so any downtime will negatively impact us.

            Rick Westbrock added a comment - Moving the deployments "out of business hours where possible" does not help us at all because we have orchestration jobs that run every hour so any downtime will negatively impact us.

            We can sometimes go for weeks with no failures in our scheduled orchestration jobs but at other times (like that past week or so) we are getting hit by this several times per week (four times in the last five days in fact). Since it is acknowledged as a bug I don't understand why it is still "gathering interest", I would expect the team to figure out how to do non-disruptive deployments ASAP.

            When we first encountered this bug last year I modified our orchestration to first call the list object schema API then wait 30 seconds since I was advised by Support that this would work around the bug but that has not been the case lately, we end up having to rerun the job later. It is also annoying that the outages are not in roughly the same time window every day, sometimes they are very early morning PST and sometimes a few hours before midnight.

            Rick Westbrock added a comment - We can sometimes go for weeks with no failures in our scheduled orchestration jobs but at other times (like that past week or so) we are getting hit by this several times per week (four times in the last five days in fact). Since it is acknowledged as a bug I don't understand why it is still "gathering interest", I would expect the team to figure out how to do non-disruptive deployments ASAP. When we first encountered this bug last year I modified our orchestration to first call the list object schema API then wait 30 seconds since I was advised by Support that this would work around the bug but that has not been the case lately, we end up having to rerun the job later. It is also annoying that the outages are not in roughly the same time window every day, sometimes they are very early morning PST and sometimes a few hours before midnight.

            Once more, I am puzzled by the fact that this is an issue. Assets are a Premium Feature and Enterprise-grade customers tend to use them (or they are even required to fulfill all their requirements). In my particular case, the support runs 24/7 and the JSM instance is highly dependent on assets. Whilst in "usual" user experience, this is an annoyance, with automations this is a no-go. The automation fails in the middle of the night, causing tickets to be stuck as the trigger passed, and the automation fails for no reason.

            Birger Robrecht [avono AG] added a comment - Once more, I am puzzled by the fact that this is an issue. Assets are a Premium Feature and Enterprise-grade customers tend to use them (or they are even required to fulfill all their requirements). In my particular case, the support runs 24/7 and the JSM instance is highly dependent on assets. Whilst in "usual" user experience, this is an annoyance, with automations this is a no-go. The automation fails in the middle of the night, causing tickets to be stuck as the trigger passed, and the automation fails for no reason.

            Hello Justin, 

            Do you have any updates on when the planned downtime will occur for each region?

            Best regards, 

            Dimitris 

            Dimitrios Moschovitis added a comment - Hello Justin,  Do you have any updates on when the planned downtime will occur for each region? Best regards,  Dimitris 

            Sean Brewington added a comment - - edited

            Hi Justin,

            Thanks for update.  I wanted to mention though on point:

             

            "Moving the regular downtime windows to be region specific"

             

            Our company accepts global JSM portal requests so there is no downtime that can really work for us.  But hopefully minimizing downtime will help.  It would be preferred to have no downtime, or be able to re-trigger or re-try any Automation Rules somehow after a delay.

             

            Thanks,

            Sean

             

            Sean Brewington added a comment - - edited Hi Justin, Thanks for update.  I wanted to mention though on point:   "Moving the regular downtime windows to be region specific"   Our company accepts global JSM portal requests so there is no downtime that can really work for us.  But hopefully minimizing downtime will help.  It would be preferred to have no downtime, or be able to re-trigger or re-try any Automation Rules somehow after a delay.   Thanks, Sean  

            Hi all,

            Is there any specific ETA for the planned downtime per region?

            Until then, it would be beneficial for deployments to happen at times other than xx:00 and xx:30 hours, as these are the available options for scheduled automation.

            Please, consider this as a temporary workaround!

            Thank you,

            Dimitris

            Dimitrios Moschovitis added a comment - Hi all, Is there any specific ETA for the planned downtime per region? Until then, it would be beneficial for deployments to happen at times other than xx:00 and xx:30 hours, as these are the available options for scheduled automation. Please, consider this as a temporary workaround! Thank you, Dimitris

              7260236ab89d Justin King
              0a9ae468a560 Marcus Fridén
              Affected customers:
              83 This affects my team
              Watchers:
              97 Start watching this issue

                Created:
                Updated: