Uploaded image for project: 'Analytics'
  1. Analytics
  2. ANALYTICS-154

Ability to force update/control the data freshness in Data lake

    • Our product teams collect and evaluate feedback from a number of different sources. To learn more about how we use customer feedback in the planning process, check out our new feature policy.

      User Problem

      Currently, the data freshness takes at least 30 minutes to sync data and for certain tables, it takes more than 30 minutes.

      I would like to find a way to schedule or force update the data in Data Lake. 

      Suggested Solutions

      An option in Data lake to force update the data

      Current Workarounds

      None available.

            [ANALYTICS-154] Ability to force update/control the data freshness in Data lake

            Jennifer Luo added a comment -

            Our dashboards are used frequently during meetings throughout the week but if the data is a few hours out of date, it means any issues raised within 3 hours of the meeting (which is very common and expected) will not show up, causing delays and confusion during presentation.

            Please revisit this - there are 3rd party applications in the marketplace that offer force updates and better data freshness than Atlassian, so why would people want to move to Atlassian Analytics?

            Jennifer Luo added a comment - Our dashboards are used frequently during meetings throughout the week but if the data is a few hours out of date, it means any issues raised within 3 hours of the meeting (which is very common and expected) will not show up, causing delays and confusion during presentation. Please revisit this - there are 3rd party applications in the marketplace that offer force updates and better data freshness than Atlassian, so why would people want to move to Atlassian Analytics?

            Just today, we had to skip a fairly important agenda topic because our dashboards didn't refresh. It was wildly out of date (even though the last full refresh was within 24 hours), and therefore, this timely urgent and important topic was missed. 

            Ashley Fraser added a comment - Just today, we had to skip a fairly important agenda topic because our dashboards didn't refresh. It was wildly out of date (even though the last full refresh was within 24 hours), and therefore, this timely urgent and important topic was missed. 

            Ollie Guan added a comment -

            The inconsistency of data refresh cycle in Data Lake and Dashboard settings has caused us confusion. How long do we have to wait to get a dashboard with all the updated data?

            https://support.atlassian.com/analytics/docs/understand-the-data-model-of-the-atlassian-data-lake/

            https://support.atlassian.com/analytics/docs/dashboard-settings/

            Ollie Guan added a comment - The inconsistency of data refresh cycle in Data Lake and Dashboard settings has caused us confusion. How long do we have to wait to get a dashboard with all the updated data? https://support.atlassian.com/analytics/docs/understand-the-data-model-of-the-atlassian-data-lake/ https://support.atlassian.com/analytics/docs/dashboard-settings/

            Seth Miller added a comment - - edited

            I support this feature request. My organization would like to use Analytics to produce regular reports for senior management. We need to be able to make changes to data that goes into the reports toward the end of each reporting cyle. Waiting 3-6 hours for changes to custom data makes the use of Analytics for this purpose prohibitive. We would not be able to get the reports out on time. My organization will have to find another tool.

            Seth Miller added a comment - - edited I support this feature request. My organization would like to use Analytics to produce regular reports for senior management. We need to be able to make changes to data that goes into the reports toward the end of each reporting cyle. Waiting 3-6 hours for changes to custom data makes the use of Analytics for this purpose prohibitive. We would not be able to get the reports out on time. My organization will have to find another tool.

            The architecture of how the data flows from products to the data lake does not really allow us to implement this.

            Kevin Minnick added a comment - The architecture of how the data flows from products to the data lake does not really allow us to implement this.

            I strongly support the feature request for enabling a force data update or scheduling specific update times in the Data Lake. Our team relies on the Data schema in Atlassian Analytics for critical internal service reviews, which are held weekly at 10 am on Mondays. The current data refresh schedule, which initiates around 10 am, significantly impacts our ability to review the most up-to-date data pertaining to the previous week's activities. This timing misalignment hinders our decision-making processes and the effectiveness of these meetings.

            Given the inflexibility of rescheduling these reviews due to calendar constraints, an ideal solution would be to adjust the data refresh timing to ensure data is refreshed more frequently. This information will allow us to better plan our meetings and manage expectations regarding data availability.

            Implementing a feature to either force a data update or to schedule updates at specified times, or with more frequency, would not only resolve our immediate challenge but would also provide greater flexibility and efficiency for all users relying on timely data for decision-making processes. Thank you for considering this request.

            Luke Collins added a comment - I strongly support the feature request for enabling a force data update or scheduling specific update times in the Data Lake. Our team relies on the Data schema in Atlassian Analytics for critical internal service reviews, which are held weekly at 10 am on Mondays. The current data refresh schedule, which initiates around 10 am, significantly impacts our ability to review the most up-to-date data pertaining to the previous week's activities. This timing misalignment hinders our decision-making processes and the effectiveness of these meetings. Given the inflexibility of rescheduling these reviews due to calendar constraints, an ideal solution would be to adjust the data refresh timing to ensure data is refreshed more frequently. This information will allow us to better plan our meetings and manage expectations regarding data availability. Implementing a feature to either force a data update or to schedule updates at specified times, or with more frequency, would not only resolve our immediate challenge but would also provide greater flexibility and efficiency for all users relying on timely data for decision-making processes. Thank you for considering this request.

             

            Hi Atlassian Team,

            I am Shankar Nulu from Mondelez International and I have raised support ticket AVP-938 but I didn't get the proper clarification but Atlassian raised Internal tickets (ANALYTICS-128 & https://jira.atlassian.com/browse/ANALYTICS-154)  but after that I don't have any response back.

            Can you please clarify me below my queries:

            1. I have a question regarding the data lake refresh frequency in Atlassian Analytics (https://mdlzdevopsanalytics.atlassian.net/analytics)
            2. I would like to understand how frequently the data lake is updated from the Jira instances / sites. Specifically, I am interested in knowing the timings and intervals at which the data lake is refreshed with updates from our Jira instances to the Analytics Data lake?
            3. Could you provide information on the typical schedule or any configurable settings related to the data lake refresh process? Currently I am setting up refresh element -  

              Data freshness  = 1 hour & Check cache every* = 30mins. Dashboard says that 1 day ago refreshed or 2 days ago or 21hrs ago chart updated in the top right corner

            1. How can we get the proper date and time when was chart got updated and some cases even thought data chart updated but data not coming ( we can find in the Jira site issue but not reflecting the dashboard. Dashboard users are asking to know what is the refresh timings for this site. How frequently data lake will be updated including exact time and how long each data lake sync takes place?

             

            Thanks & Regards,

            Shankar Nulu

            Shankar Nulu added a comment -   Hi Atlassian Team, I am Shankar Nulu from Mondelez International and I have raised support ticket AVP-938 but I didn't get the proper clarification but Atlassian raised Internal tickets ( ANALYTICS-128 & https://jira.atlassian.com/browse/ANALYTICS-154 )  but after that I don't have any response back. Can you please clarify me below my queries: I have a question regarding the data lake refresh frequency in Atlassian Analytics ( https://mdlzdevopsanalytics.atlassian.net/analytics) I would like to understand how frequently the data lake is updated from the Jira instances / sites. Specifically, I am interested in knowing the timings and intervals at which the data lake is refreshed with updates from our Jira instances to the Analytics Data lake? Could you provide information on the typical schedule or any configurable settings related to the data lake refresh process? Currently I am setting up refresh element -   Data freshness  = 1 hour & Check cache every* = 30mins. Dashboard says that 1 day ago refreshed or 2 days ago or 21hrs ago chart updated in the top right corner How can we get the proper date and time when was chart got updated and some cases even thought data chart updated but data not coming ( we can find in the Jira site issue but not reflecting the dashboard. Dashboard users are asking to know what is the refresh timings for this site. How frequently data lake will be updated including exact time and how long each data lake sync takes place?   Thanks & Regards, Shankar Nulu

              83980d655b1d Kevin Minnick
              0a2a7c90f190 Agaci Avinas
              Votes:
              35 Vote for this issue
              Watchers:
              31 Start watching this issue

                Created:
                Updated: