-
Bug
-
Resolution: Fixed
-
Low (View bug fix roadmap)
-
5.2.6
-
None
-
5.02
-
Symptoms
When synchronising a directory from a user directory, it fails, with an error like the below:
2013-02-20 14:59:35,394 QuartzWorker-0 ERROR ServiceRunner [atlassian.crowd.directory.DbCachingDirectoryPoller] Error occurred while refreshing the cache for directory [ 10001 ]. com.atlassian.jira.crowd.embedded.ofbiz.db.DataAccessException: org.ofbiz.core.entity.GenericEntityException: while inserting: [GenericEntity:Membership][id,10017][membershipType,GROUP_USER][lowerParentName,jira-users][parentId,10013][childId,10014][childName,dmason][lowerChildName,dmason][directoryId,10001][parentName,jira-users] (SQL Exception while executing the following:INSERT INTO public.cwd_membership (ID, parent_id, child_id, membership_type, group_type, parent_name, lower_parent_name, child_name, lower_child_name, directory_id) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) (ERROR: duplicate key value violates unique constraint "uk_mem_parent_child_type" Detail: Key (parent_id, child_id, membership_type)=(10013, 10014, GROUP_USER) already exists.)) at com.atlassian.jira.crowd.embedded.ofbiz.db.OfBizHelper.createValue(OfBizHelper.java:167) at com.atlassian.jira.crowd.embedded.ofbiz.OfBizInternalMembershipDao.createMembership(OfBizInternalMembershipDao.java:101) at com.atlassian.jira.crowd.embedded.ofbiz.OfBizInternalMembershipDao.addUserToGroup(OfBizInternalMembershipDao.java:94) at com.atlassian.jira.crowd.embedded.ofbiz.OfBizDelegatingMembershipDao.addUserToGroup(OfBizDelegatingMembershipDao.java:78) at com.atlassian.crowd.directory.AbstractInternalDirectory.addUserToGroup(AbstractInternalDirectory.java:695) at com.atlassian.crowd.directory.DbCachingRemoteChangeOperations.addUserToGroup(DbCachingRemoteChangeOperations.java:1151) at com.atlassian.crowd.directory.DirectoryCacheImplUsingChangeOperations.addUserToGroup(DirectoryCacheImplUsingChangeOperations.java:187) at com.atlassian.crowd.directory.ldap.cache.EventTokenChangedCacheRefresher.synchroniseChanges(EventTokenChangedCacheRefresher.java:115) at com.atlassian.crowd.directory.DbCachingRemoteDirectory.synchroniseCache(DbCachingRemoteDirectory.java:609) at com.atlassian.crowd.manager.directory.DirectorySynchroniserImpl.synchronise(DirectorySynchroniserImpl.java:63) at com.atlassian.crowd.directory.DbCachingDirectoryPoller.pollChanges(DbCachingDirectoryPoller.java:50) at com.atlassian.crowd.manager.directory.monitor.poller.DirectoryPollerJob.execute(DirectoryPollerJob.java:34) at org.quartz.core.JobRunShell.run(JobRunShell.java:195) at com.atlassian.multitenant.quartz.MultiTenantThreadPool$MultiTenantRunnable.run(MultiTenantThreadPool.java:72) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:520) Caused by: org.ofbiz.core.entity.GenericEntityException: while inserting: [GenericEntity:Membership][id,10017][membershipType,GROUP_USER][lowerParentName,jira-users][parentId,10013][childId,10014][childName,dmason][lowerChildName,dmason][directoryId,10001][parentName,jira-users] (SQL Exception while executing the following:INSERT INTO public.cwd_membership (ID, parent_id, child_id, membership_type, group_type, parent_name, lower_parent_name, child_name, lower_child_name, directory_id) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?) (ERROR: duplicate key value violates unique constraint "uk_mem_parent_child_type" Detail: Key (parent_id, child_id, membership_type)=(10013, 10014, GROUP_USER) already exists.)) at org.ofbiz.core.entity.GenericDAO.singleInsert(GenericDAO.java:136) at org.ofbiz.core.entity.GenericDAO.insert(GenericDAO.java:101) at org.ofbiz.core.entity.GenericHelperDAO.create(GenericHelperDAO.java:64) at org.ofbiz.core.entity.GenericDelegator.create(GenericDelegator.java:487) at org.ofbiz.core.entity.GenericValue.create(GenericValue.java:98) at com.atlassian.jira.crowd.embedded.ofbiz.db.OfBizHelper.createValue(OfBizHelper.java:162) ... 14 more
Cause
A group membership is being synchronised from the remote directory which already exists, and JIRA is attempting to add this. There are a few reasons why the remote directory would synchronise such memberships, including but not limited to the following known bugs:
| |
If a user exists in two directories in an application, and is added to a group in one directory that it was already in the other, clients get duplicate key errors |
Workaround
Flushing the cache of the directory will stop the error from happening, and cause the directory to synchronise fully, until the next time a duplicate group membership is sent from the remote directory. The simplest way of forcing this is:
- Disable the directory
- Enable the directory
For the specific causes there may be workarounds in the linked bug tickets.
[JRASERVER-31793] JIRA attempts to apply group memberships when synchronising from a remote directory even if they already exist, causing a duplicate key violation
A fix has been pushed to both stable and master. Ready for QA now.
A more comprehensive fix as detailed by CWD-3188. Upgrading stable and master now to get these fixes.
It turns out that changing the business logic of incremental sync events is a bit trickier than we initially thought due to some shadowed membership scenarios that are incidentally covered by the current post-processing of user-group membership deletion done in the server.
This will need to be looked at more closely.
In any case, AbstractInternalDirectory should never attempt to add a duplicate membership row as detailed by CWD-3182. I have pushed a fix to that problem to an issue branch of 2_4_x_stable.
I have finally finished investigating this issue. In order to reproduce the problem, it is neccesary to setup a JIRA Internal Directory and a Remote Crowd Directory. Incremental sync needs to be on.
Steps are as follow:
1.- Set up JIRA and add a user e.g. captain.planet.
2.- Set up a Crowd instance and add the captain.planet user.
3.- Add a jira-users group to Crowd and add the captain.planet user to it.
4.- Configure a JIRA Application in Crowd with the details of the JIRA instance setup in step 1. Make sure that this application includes the captain.planet user added in the previous step.
5.- Add a Remote Crowd Directory to the JIRA instance pointing to the Crowd instance setup in the previous steps. Make sure incremental synchronisation is on.
6.- A successful full synchronisation should be performed after the directory is added.
7.- Go to the Admin Console for Crowd and remove the captain.planet user from the jira-users group. Immediately, add the user back to the jira-users group.
8.- Go to the User Directories page in JIRA and Synchronise the Remote Crowd Directory.
9.- [BUG] The synchronisation fails and a stack trace similar to the one in this bug report will appear in the jira logs.
Root Cause:
-----------
The bug is caused by the incremental sync processing done in the crowd server. When JIRA attempts to retrieve all events after the last succesfull syncronisation, the removal of a group membership will not be sent correctly if the user was added back to that group later on in time. However, an event will still be sent for adding the user back into the group.
In the example from the reproduction steps, this will cause JIRA to attempt adding the user to the jira-users group. However, since crowd ommitted sending the intermediate membership removal. A membership record will already exist prior to the sync for that user and the jira-users group so attempting to add the membership back again will cause a duplicate primary key exception.
I have talked to jwalton about this issue and the fix is to send the intermediate membership removal in the list of incremental sync events.
jwalton has asked me to write a functional test in crowd for this scenario, so we can proceed to do the fix. I will start working on that now.
This bug is breaking crowd synchronization and people are left outside, that's clearly not a Minor issue and it has important effects on LIVE production systems.