-
Suggestion
-
Resolution: Fixed
-
None
Issue Summary
Data Shares do not offer change logs beyond 2 days due to a vacuum job that deletes all past data beyond a couple of days. While reading data from a timestamp from shared tables, you will experience 404 error if:
- batch/streaming data from a timestamp that is 2+ days ago
- batch/streaming job has large lag
Steps to Reproduce
- Use the table_changes function to attempt to see changes beyond 2 days old.
- This query will return a 404 error.
Expected Results
End users expect to be able to run change log queries beyond 2 days and for that data to be retained longer. It is expected that there is an expiration policy on CDC history, however, two days is a tight window and it would be valuable to extend the availability of this data to at least a week.
Actual Results
While reading data from a timestamp from shared tables, you will experience 404 error if:
- batch/streaming data from a timestamp that is 2+ days ago
- batch/streaming job has large lag
Workaround
Currently there is no known workaround for this behavior. A workaround will be added here when available
Hello,
Versions are now available for 7 days.
However, we still recommend using the row_refreshed_at field, which is always available.
Check this article to know more: Extract the latest changes from data shares
Cheers,
Claire
PM Atlassian Data Lake