Issue Summary

As an Assets Admin / Schema manager, a user may decide to lower the Cardinality Max value of an attribute.

In JSM V>9.16.1 there is a warning note on the Cardinality settings, but not in 9.12.12:

At this point, there might still be objects where this attribute contains a number of values, exceeding the new cardinality max value. Visiting these objects will show all exceeding values. Trying to Edit these objects will randomly remove values (random to the User!)

If objects with exceeding values are not edited, they will still show all their values.

The next time the instance is restarted or, the next Assets re-index, all exceeding values will be deleted from the database!

There will be no indication for the values removal anywhere.

This is a change of behavior from previous versions, where it was up to the customer to manually delete excessive values. Attempting to save such an object will show an error message under the attribute, indicating Cardinality violation.

The only mention for this is in the Jira Audit Log, IF the App area is increased to FULL, will be for a user changing the Attribute's cardinality settings.

The removal of values will not be reflected in the Assets audit log, Object history or other.

When re-index is executed (e.g. a week later) - the logged WARN message will indicate that Object X has duplicated values, and these were removed from the db

There is no indication which values were removed - "oldest value" can only be identified in the db.

Excessive values for older object might contain valid information for a user, who needs to keep them in place, unless removed manually.

An Indexing function should not DELETE from the db.

Steps to Reproduce

1. Create two object types: OTA, OTB
2. Create a few OTB Objects: B1, B2, B3, B4
3. In OTA's Attributes tab, add an attribute to reference OTB, with Cardinality max set to unlimited (or any value > 1)
4. Create two OTA object: A1, A2 and set each to reference the objects:  B2, B4, B1, B3 (add objects in a random order)
5. In OTA's Attributes tab - change cardinality max of the attribute back to 1
6. Switch back to objects view - note both A1 and A2 are still showing all references correctly.
7. Try to edit A2 - note that as soon as you enter the edit screens, some values disappear from the screen (exit the screen with the Cancel button - the values are still there)
8. Re-index Assets and monitor the Jira log

Expected Results

NO values are deleted from the DB due to re-indexing.

If ever values/data is being permanently deleted, it will be noted and requires a review/confirmation.

Data removal should be clearly and correctly recorded.

Actual Results

Objects will lose excessive values some time after the cardinality change was made, and values will randomly disappear, with no visible logic and no trace. (User may not want values removed automatically for historical reasons)

Log will show misleading messages about Duplicates during indexing:

2024-09-06 16:04:52,516-0400 insight-InsightThreadGroup-worker-thread-14 WARN userX     [c.r.j.p.i.s.c.dal.impl.ObjectDalImpl] Found duplicate values for object with id 1367397, deleting them from the database
2024-09-06 16:04:53,078-0400 insight-InsightThreadGroup-worker-thread-14 WARN userX     [c.r.j.p.i.s.c.dal.impl.ObjectDalImpl] Duplicates resolved. Reloading objects from database - [1367397]
2024-09-06 16:04:53,369-0400 insight-InsightThreadGroup-worker-thread-14 WARN userX     [c.r.j.p.i.s.c.dal.impl.ObjectDalImpl] Sending cache update messages for objects - [1367397]

If the change affects many objects - it may affect performance, e.g. a slower startup of the application, due to the deletion operation.

Previously, excessive values were kept to the user's consideration.

Workaround

Add the following Dark Feature to disable duplicates handling during reindexing:

assets.index.dedupe.attributes.disabled

Do note, enabling this feature flag might result in duplicate attribute values after imports in the future.

Versions Tested

Please confirm all versions that have been tested for this issue, and indicate whether the tested version is affected or not affected, below: