-
Bug
-
Resolution: Duplicate
-
Medium
-
20
-
Severity 3 - Minor
-
33
-
-
Issue Summary
Insight is unable on Wednesday at midnight UTC due to a routine node cycling process that happens weekly in all regions.
Background -
Assets stores all the objects information in memory so that the information is readily available when a customer performs a query on the UI and enables us to respond timely to the requests. Every time node cycling happens, the cache is cleared and reloaded from database for meeting performance. During the time cache is reloaded, the system responds a 429 to make sure no requests are returned incorrectly.
The time to reload the cache is equivalent to a down time for customers since the system cannot respond to requests in the timespan. On an average this reload only takes a couple of seconds, but for customers with a large number of data it can be expected to take up to 15-20 min of down time.
In that time, the system gets ready to serve requests, and requests are then handled gracefully.
We are actively working to decrease this time to the minimum we can at our earliest.
Steps to Reproduce
Access Insight on Wednesday at midnight UTC.
Expected Results
Insight doesn't experience downtime impacting customers. The maintenance window for the node cycling process needed to be better aligned with the customer region like between 3 am - 5 am for customers depending on location, to avoid downtime.
Actual Results
Insight is unavailable for a couple of minutes interrupting the customer's work.
Workaround
Currently, there is no known workaround for this behavior. A workaround will be added here when available
- duplicates
-
-
- In Progress
-
- blocks
-
ACE-3559 You do not have permission to view this issue
-
PCS-178546 You do not have permission to view this issue
[JSDCLOUD-11834] Unavailability of Insight on Wednesday at midnight UTC
Periodically, one of our Automation that it is fetching data from Assets to Jira tickets it is failing without any real reason.
According to the Atlassian support team that was investigating the reason, it is related to this bug.
Please, fix this since it is failing daily.
Please fix it asap, sometimes it's really frustrating and downtime occurs on our main shift.
Noticed by users again today during the restart... please consider setting the downtime outside of business hours around the world!
Automations using Insight which happened during this time also failed - this is very impactful to our BAU requirements
We will be working on a solution in the upcoming innovation sprint to improve the customer experience (i.e. minimise downtime) during node cycling. It would be good to understand how many customers this is affecting, hence the ticket status is in "gathering impact".
Inisight is always required during business hours as part of CMDB integral to our ITSM. Currently this happens mid morning in Australian time.
Atlassian willl need to cater for a global audience and do any service impacting maintenance to out of business hours!
this is definitely something that needs to be fixed ASAP... makes no sense to have downtimes in the middle of business hours!
Hello!
This bug is a duplicate of https://jira.atlassian.com/browse/JSDCLOUD-12246 and will be closed. All future updates will be made on https://jira.atlassian.com/browse/JSDCLOUD-12246 so please directly your comments and feedback there.
Thanks very much.
Justin King,
Product Manager, Jira Service Management