When syncing with external directories Crowd is mistakenly including inactive users that have been deleted externally

Issue Summary

When connecting Crowd to an external user directory such as LDAP it keeps track of users that have been deleted from LDAP but have relating content in our applications- these are known as inactive users. An example database entry will resemble the following in the cwd_user table:

It appears when users to be deleted are calculated, then all users(active + inactive) from database are taken into account, but only active should be.

class UsnChangedCacheRefresher
method validateAndReturnGuidsToAddAndDelete(DirectoryCache directoryCache)

bug:
final Set<String> guidsInCache = ImmutableSet.copyOf(directoryCache.getAllUserGuids());

Steps to Reproduce

1. Connect an Atlassian application to external user directory
2. Create content for a user
3. Remove this user from the external user directory

Expected Results

Crowd will mark this user as inactive and never reference this user again while syncing with external user directories

Actual Results

Crowd continues using the inactive / externally deleted user when calculating users it needs to delete. As this behavior accumulates in larger environments it will lead to longer synchronization times

The sync logs from atlassian-jira.log for example will resemble the following:

2020-01-23 16:37:37,499 Caesium-1-2 INFO ServiceRunner     [c.a.crowd.directory.DbCachingRemoteChangeOperations] deleted [ 1 ] users in [ 21ms ]
2020-01-23 16:41:08,432 Caesium-1-4 INFO ServiceRunner     [c.a.crowd.directory.DbCachingRemoteChangeOperations] deleted [ 1 ] users in [ 10ms ]
2020-01-23 16:41:21,206 Caesium-1-2 INFO ServiceRunner     [c.a.crowd.directory.DbCachingRemoteChangeOperations] deleted [ 1 ] users in [ 11ms ]

Notice how Crowd is attempting to "delete" the same user ( depicted above ) over and over again

Workaround

Currently there is no known workaround for this behavior. A workaround will be added here when available