Uploaded image for project: 'Analytics'
  1. Analytics
  2. ANALYTICS-352

Offer change log queries beyond 2 days for Data Shares

    • Our product teams collect and evaluate feedback from a number of different sources. To learn more about how we use customer feedback in the planning process, check out our new feature policy.

      Issue Summary

      Data Shares do not offer change logs beyond 2 days due to a vacuum job that deletes all past data beyond a couple of days. While reading data from a timestamp from shared tables, you will experience 404 error if:

      • batch/streaming data from a timestamp that is 2+ days ago
      • batch/streaming job has large lag

      Steps to Reproduce

      1. Use the table_changes function to attempt to see changes beyond 2 days old. 
      2. This query will return a 404 error. 

      Expected Results

      End users expect to be able to run change log queries beyond 2 days and for that data to be retained longer. It is expected that there is an expiration policy on CDC history, however, two days is a tight window and it would be valuable to extend the availability of this data to at least a week.

      Actual Results

      While reading data from a timestamp from shared tables, you will experience 404 error if:

      • batch/streaming data from a timestamp that is 2+ days ago
      • batch/streaming job has large lag

      Workaround

      Currently there is no known workaround for this behavior. A workaround will be added here when available

            [ANALYTICS-352] Offer change log queries beyond 2 days for Data Shares

            Claire Gayrard added a comment -

            Hello,

            Versions are now available for 7 days.
            However, we still recommend using the row_refreshed_at field, which is always available.

            Check this article to know more: Extract the latest changes from data shares

             

            Cheers,
            Claire

            PM Atlassian Data Lake

            Claire Gayrard added a comment - Hello, Versions are now available for 7 days. However, we still recommend using the row_refreshed_at field, which is always available. Check this article to know more: Extract the latest changes from data shares   Cheers, Claire PM Atlassian Data Lake

              7f311a70eaad Claire Gayrard
              2cfa2301498f Skyler Ataide
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Created:
                Updated:
                Resolved: