Unavailability of Insight on Wednesday at midnight UTC

Issue Summary

Insight is unable on Wednesday at midnight UTC due to a routine node cycling process that happens weekly in all regions.

Background -

Assets stores all the objects information in memory so that the information is readily available when a customer performs a query on the UI and enables us to respond timely to the requests. Every time node cycling happens, the cache is cleared and reloaded from database for meeting performance. During the time cache is reloaded, the system responds a 429 to make sure no requests are returned incorrectly.

The time to reload the cache is equivalent to a down time for customers since the system cannot respond to requests in the timespan. On an average this reload only takes a couple of seconds, but for customers with a large number of data it can be expected to take up to 15-20 min of down time.

In that time, the system gets ready to serve requests, and requests are then handled gracefully.

We are actively working to decrease this time to the minimum we can at our earliest.

Steps to Reproduce

Access Insight on Wednesday at midnight UTC.

Expected Results

Insight doesn't experience downtime impacting customers. The maintenance window for the node cycling process needed to be better aligned with the customer region like between 3 am - 5 am for customers depending on location, to avoid downtime.

Actual Results

Insight is unavailable for a couple of minutes interrupting the customer's work.

Workaround

Currently, there is no known workaround for this behavior. A workaround will be added here when available