Uploaded image for project: 'Crowd Data Center'
  1. Crowd Data Center
  2. CWD-5182

Sync Failures due to duplicated External IDs

XMLWordPrintable

      Summary

      Full Synchronisation in between Crowd and any of the Atlassian Applications fails due to multiple users with the same External ID.

      Steps to Replicate

      It is unclear how Crowd duplicate External_IDs in its database without manual intervention, but, to replicate this problem we should:

      1. Select one of the Atlassian applications and connect it to Crowd as the User Management platform. For this scenario, Confluence was chosen.
      2. With that completed, add two users to Crowd, specifically to the directory where Confluence is reading users from. For this scenario, we added User1 and User2.
      3. Open Crowd database and run the following query to set both users with the same External ID: 
        update cwd_user set external_id = (select external_id from cwd_user where lower_user_name = 'user1')
where lower_user_name = 'user2';
      1. With that completed, run a Synchronisation through Confluence User Directory screen, it will fail.

      Log Evidence

      The synchronisation fails due to the following error - Multiple entries with the same key:

      2018-06-11 15:08:35,851 ERROR [Caesium-1-4] [atlassian.crowd.directory.DbCachingDirectoryPoller] pollChanges Error occurred while refreshing the cache for directory [ 950274 ].
java.lang.IllegalArgumentException: Multiple entries with same key: 458753:3bfe6523-ba06-471a-bb77-ad2a623dc7eb=com.atlassian.crowd.model.user.InternalUser@4577c9c[id=1081348,name=user,createdDate=2018-06-11 15:07:22.002,updatedDate=2018-06-11 15:07:22.002,active=true,emailAddress=user1@userinc.com,firstName=User,lastName=Uno,displayName=User Uno,credential=com.atlassian.crowd.embedded.api.PasswordCredential@d11ef10[credential=********,encryptedCredential=true],lowerName=user1,lowerEmailAddress=user1@userinc.com,lowerFirstName=user,lowerLastName=uno,lowerDisplayName=user uno,directoryId=950274,externalId=458753:3bfe6523-ba06-471a-bb77-ad2a623dc7eb] and 458753:3bfe6523-ba06-471a-bb77-ad2a623dc7eb=com.atlassian.crowd.model.user.InternalUser@6a222bb[id=1081347,name=dude,createdDate=2018-06-11 15:07:22.002,updatedDate=2018-06-11 15:07:22.002,active=true,emailAddress=user2@userinc.com,firstName=User,lastName=Dos,displayName=User Dos Uno,credential=com.atlassian.crowd.embedded.api.PasswordCredential@39b45c78[credential=********,encryptedCredential=true],lowerName=user2,lowerEmailAddress=user2@userinc.com,lowerFirstName=user,lowerLastName=dos,lowerDisplayName=user dos,directoryId=950274,externalId=458753:3bfe6523-ba06-471a-bb77-ad2a623dc7eb]
	at com.google.common.collect.ImmutableMap.checkNoConflict(ImmutableMap.java:150)
	at com.google.common.collect.RegularImmutableMap.checkNoConflictInBucket(RegularImmutableMap.java:104)
	at com.google.common.collect.RegularImmutableMap.<init>(RegularImmutableMap.java:70)
	at com.google.common.collect.ImmutableMap$Builder.build(ImmutableMap.java:254)
	at com.atlassian.crowd.directory.DbCachingRemoteChangeOperations.mapUsersByExternalId(DbCachingRemoteChangeOperations.java:1194)
	at com.atlassian.crowd.directory.DbCachingRemoteChangeOperations.getUsersToAddAndUpdate(DbCachingRemoteChangeOperations.java:1106)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:302)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
	at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:99)
	at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:281)
	at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
	at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
	at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:208)
	at com.atlassian.crowd.directory.$Proxy2496.getUsersToAddAndUpdate(Unknown Source)
	at com.atlassian.crowd.directory.DirectoryCacheImplUsingChangeOperations.addOrUpdateCachedUsers(DirectoryCacheImplUsingChangeOperations.java:55)
	at com.atlassian.crowd.directory.ldap.cache.RemoteDirectoryCacheRefresher.synchroniseAllUsers(RemoteDirectoryCacheRefresher.java:88)
	at com.atlassian.crowd.directory.ldap.cache.AbstractCacheRefresher.synchroniseAll(AbstractCacheRefresher.java:56)
	at com.atlassian.crowd.directory.ldap.cache.EventTokenChangedCacheRefresher.synchroniseAll(EventTokenChangedCacheRefresher.java:69)
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.synchroniseCache(DbCachingRemoteDirectory.java:1186)
	at com.atlassian.crowd.manager.directory.DirectorySynchroniserImpl.synchronise(DirectorySynchroniserImpl.java:74)
	at com.atlassian.crowd.directory.DbCachingDirectoryPoller.pollChanges(DbCachingDirectoryPoller.java:50)
	at com.atlassian.crowd.manager.directory.monitor.poller.DirectoryPollerJobRunner.runJob(DirectoryPollerJobRunner.java:96)

      Steps to Diagnose and Workaround

      In order to confirm that your instance is indeed affected by this issue, you need to see above stack trace in your logs and the query below should return results - Don't forget to replace the ID of the affected directory below:

      SELECT id, active, user_name, external_id, directory_id FROM cwd_user WHERE external_id IN 
(SELECT external_id FROM cwd_user WHERE directory_id = 'YOUR_DIRECTORY_ID_HERE' GROUP BY external_id HAVING COUNT(external_id) > 1) order by 4;

      In case you don't have the ID of the affected directory, run the query below:

      select * from cwd_directory;

      Once you have the results handy, you have to:

      1. Access Crowd and remove the affected users from the directory Confluence (or other application) is syncing to.
        1. If you are using a LDAP Connector (not a Delegated Directory nor a Crowd Internal Directory), you need to run the clean-up at your LDAP too or filter the duplicated users through a LDAP query.
      2. In case those users are not supposed to have access anymore, proceed to the next step. Else, re-add those users to Crowd and grant them access to the directory Confluence (or other application) is syncing to. This will generate new External IDs for them.
      3. Synchronise directories through Confluence (or other application) user directory screen.

      Suggestion

      Perhaps this bug can be fixed through adding a constraint under cwd_user table that specifies one external_id per directory_id. There's a unique key constraint currently that defines one lower_user_name per directory. This Feature request is logged here: https://jira.atlassian.com/browse/CWD-3882

      Workaround

      1. As a possible, examine the duplicate records returned by the diagnostic query above and determine which user_name entry that you want to be used for the user.
      2. For the other record of the user, set the external_id to a text entry such as 'invalid'. For example: 
        update cwd_user set external_id = 'invalid' where id = <the record ID of the entry with the user_name that shouldn't be used>;

            [CWD-5182] Sync Failures due to duplicated External IDs

            Rudy Slaiby made changes -
            Remote Link New: This issue links to "Page (Confluence)" [ 637245 ]
            Pawel Gruszczynski (Inactive) made changes -
            Fix Version/s New: 4.4.0 [ 95591 ]
            Fix Version/s New: 4.4.1 [ 97911 ]
            Pawel Gruszczynski (Inactive) made changes -
            Resolution New: Fixed [ 1 ]
            Status Original: In Progress [ 3 ] New: Closed [ 6 ]
            Pawel Gruszczynski (Inactive) made changes -
            Status Original: Long Term Backlog [ 12073 ] New: In Progress [ 3 ]
            SET Analytics Bot made changes -
            UIS Original: 419 New: 401
            SET Analytics Bot made changes -
            UIS Original: 461 New: 419
            SET Analytics Bot made changes -
            UIS Original: 503 New: 461
            SET Analytics Bot made changes -
            Support reference count Original: 82 New: 83
            SET Analytics Bot made changes -
            UIS Original: 441 New: 503
            SET Analytics Bot made changes -
            UIS Original: 442 New: 441

              Unassigned Unassigned
              mhorlle Marcelo Horlle
              Affected customers:
              15 This affects my team
              Watchers:
              46 Start watching this issue

                Created:
                Updated:
                Resolved: