Problem Definition

When importing objects using the AWS Insight Integration and setting the "Missing Objects" handling to "Delete," the time taken to delete missing objects can sometimes increase the overall import time significantly.

This is due to the time taken to delete object references on related objects. Before proceeding with importing the next object, the integration waits until the delete operation has completed.

Suggested Solution

Delete missing objects asynchronously from the rest of the import. Allow new objects to be imported while the existing objects are deleted either at the end of the import process or in a separate process thread.

Why this is important

When setting Duplicate Objects to Delete for an AWS import, if there are many hundreds of related objects, this can slow the import process down quite a lot. We have observed cases where the import takes 14x as long as when not deleting duplicate objects – a 30 minute import takes 7 hours.

Workaround

You can use a new attribute and automation to delete duplicate objects outside of the import:

1. Create a status attribute for each object type named "Soft Delete" with statuses (for example) "Active" and "Deleted"
2. In the object type import configuration, set Missing Objects to Update, and chose the attribute you created (Soft Delete). Set the Soft Delete status to Deleted.
3. You can later delete these objects manually by filtering for them and using a bulk delete. Alternately, you can schedule an automation rule to delete them on a set schedule.