Customer Use Case

The customer has an hourly batch process that retrieves incremental updates from Enterprise Insights using the [Date DW Modified] field to filter and fetch recently modified records.

Problem

During the two most recent Enterprise Insights releases that included database schema changes, the [Date DW Modified] timestamp was updated on approximately 3.85 million records, resulting in a significant spike in data volume. As a result, during the batch runs on the affected dates, the API calls experienced timeouts due to the unexpectedly high volume of data. It also causes old work items to show up as recently updated, leading to false positives.

Suggestion

• To prevent similar issues in the future, we recommended to the EI Engineering team that schema-related updates be tracked using a separate field or table, rather than modifying the [Date DW Modified] field used for operational data tracking.
• Alternatively, provide us with recommendations to track the changes better without relying on the [Date DW Modified] field.