Details
-
Bug
-
Resolution: Fixed
-
Medium
-
2.7, 2.8
-
None
Description
Problem summary
If a group is renamed or removed on the LDAP server in the middle of a directory sync, the entire sync will fail/terminate with an LDAP error code 32, which means an object not found. In theory this can occur in any LDAP directory; in practice, this is more likely to impact instances that are connected to larger LDAP directories that can take a long time to sync.
For our largest customers, the impact of this bug can be very profound since directory syncs typically take a long time even if successful. The longer it takes, the more likely there will be a change to a group in LDAP before the sync is finished, and therefore the more likely the sync will fail, repeatedly. This means that new LDAP data will not get cached until a long time later until Crowd is able to perform a successful sync.
Ideally, Crowd would acknowledge this situation more gracefully, and move onto the next group instead of failing the entire sync.
This also impacts all Atlassian applications that use Crowd libraries to sync to LDAP (Confluence, JIRA, Stash, etc).
Steps to reproduce
- Configure a large LDAP structure that contains a lot of groups. (For reference: my LDAP server I tested with contained over 3000 groups.)
- In Crowd, Configure a Connector to this LDAP server
- Start a sync
- Immediately change a group CN value in the LDAP server such that the DN is now changed (alternatively: delete the group altogether). This requires a bit of timing because you need to change it after Crowd finishes its initial query that asks for a listing of remote groups:
2015-03-13 19:20:52,913 CrowdUsnChangedCacheRefresher:thread-2 INFO [directory.ldap.cache.UsnChangedCacheRefresher] found [ 303 ] remote groups in [ 15014ms ]
- Crowd will eventually query for each individual group it found in the previous step. When it encounters the group you modified, the sync will fail because the DN it's looking for is not found.
Stacktrace
In the stacktrace below, you can see that Crowd is attempting to query the group object at cn=examplegroup,ou=testing,dc=example,dc=com. However, by the time it gets to this part, that group no longer exists at that DN, and therefore the LDAP server returned with an error code 32.
2015-03-13 19:21:46,588 scheduler_Worker-3 INFO [atlassian.crowd.directory.DbCachingRemoteDirectory] failed synchronisation complete for directory [ 360449 ] in [ 68690ms ] 2015-03-13 19:21:46,597 scheduler_Worker-3 ERROR [atlassian.crowd.directory.DbCachingDirectoryPoller] Error occurred while refreshing the cache for directory [ 360449 ]. org.springframework.ldap.NameNotFoundException: [LDAP: error code 32 - 0000208D: NameErr: DSID-031001CD, problem 2001 (NO_OBJECT), data 0, best match of: 'OU=testing,DC=example,DC=com' ]; nested exception is javax.naming.NameNotFoundException: [LDAP: error code 32 - 0000208D: NameErr: DSID-031001CD, problem 2001 (NO_OBJECT), data 0, best match of: 'OU=testing,DC=example,DC=com' ]; remaining name 'cn=examplegroup,ou=testing,dc=example,dc=com' at org.springframework.ldap.support.LdapUtils.convertLdapException(LdapUtils.java:183) at org.springframework.ldap.core.LdapTemplate.executeWithContext(LdapTemplate.java:820) at org.springframework.ldap.core.LdapTemplate.executeReadOnly(LdapTemplate.java:803) at org.springframework.ldap.core.LdapTemplate.lookup(LdapTemplate.java:935) at com.atlassian.crowd.directory.ldap.SpringLdapTemplateWrapper$9.timedCall(SpringLdapTemplateWrapper.java:286) at com.atlassian.crowd.directory.ldap.SpringLdapTemplateWrapper$TimedCallable.call(SpringLdapTemplateWrapper.java:124) at com.atlassian.crowd.directory.ldap.SpringLdapTemplateWrapper.invokeWithContextClassLoader(SpringLdapTemplateWrapper.java:87) at com.atlassian.crowd.directory.ldap.SpringLdapTemplateWrapper.lookup(SpringLdapTemplateWrapper.java:282) at com.atlassian.crowd.directory.RFC4519Directory.findDirectMembersOfGroup(RFC4519Directory.java:959) at com.atlassian.crowd.directory.MicrosoftActiveDirectory.findDirectMembersOfGroup(MicrosoftActiveDirectory.java:516) at com.atlassian.crowd.directory.RFC4519DirectoryMembershipsIterable$2.apply(RFC4519DirectoryMembershipsIterable.java:78) at com.atlassian.crowd.directory.RFC4519DirectoryMembershipsIterable$2.apply(RFC4519DirectoryMembershipsIterable.java:70) at com.google.common.collect.Iterators$8.next(Iterators.java:812) at com.atlassian.crowd.directory.ldap.cache.AbstractCacheRefresher.synchroniseMemberships(AbstractCacheRefresher.java:128) at com.atlassian.crowd.directory.ldap.cache.AbstractCacheRefresher.synchroniseAll(AbstractCacheRefresher.java:84) at com.atlassian.crowd.directory.ldap.cache.UsnChangedCacheRefresher.synchroniseAll(UsnChangedCacheRefresher.java:161) at com.atlassian.crowd.directory.DbCachingRemoteDirectory.synchroniseCache(DbCachingRemoteDirectory.java:1120) at com.atlassian.crowd.manager.directory.DirectorySynchroniserImpl.synchronise(DirectorySynchroniserImpl.java:76) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.transaction.interceptor.TransactionInterceptor$1.proceedWithInvocation(TransactionInterceptor.java:96) at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:260) at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy28.synchronise(Unknown Source) at com.atlassian.crowd.directory.DbCachingDirectoryPoller.pollChanges(DbCachingDirectoryPoller.java:50) at com.atlassian.crowd.manager.directory.monitor.poller.DirectoryPollerJobRunner.runJob(DirectoryPollerJobRunner.java:93) at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:135) at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:101) at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:80) at com.atlassian.scheduler.quartz1.Quartz1Job.execute(Quartz1Job.java:32) at org.quartz.core.JobRunShell.run(JobRunShell.java:223) at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549) Caused by: javax.naming.NameNotFoundException: [LDAP: error code 32 - 0000208D: NameErr: DSID-031001CD, problem 2001 (NO_OBJECT), data 0, best match of: 'OU=testing,DC=example,DC=com' ]; remaining name 'cn=examplegroup,ou=testing,dc=example,dc=com' at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3112) at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:3033) at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2840) at com.sun.jndi.ldap.LdapCtx.c_getAttributes(LdapCtx.java:1332) at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_getAttributes(ComponentDirContext.java:231) at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:139) at javax.naming.directory.InitialDirContext.getAttributes(InitialDirContext.java:152) at sun.reflect.GeneratedMethodAccessor359.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.springframework.ldap.transaction.compensating.manager.TransactionAwareDirContextInvocationHandler.invoke(TransactionAwareDirContextInvocationHandler.java:90) at com.sun.proxy.$Proxy383.getAttributes(Unknown Source) at org.springframework.ldap.core.LdapTemplate$17.executeWithContext(LdapTemplate.java:937) at org.springframework.ldap.core.LdapTemplate.executeWithContext(LdapTemplate.java:817) ... 37 more
Workarounds
- You can reduce the chance of this happening by limiting your LDAP sync scope (therefore reducing sync duration) by using a more specific DN or writing a more restrictive LDAP filter in your user directory configurations.
- Enable the Incremental Sync option if your LDAP structure supports it, to further reduce the sync time
- If you have a large LDAP structure and are unable to restrict the sync scope, we do not recommend using a Connector (syncing directory). Instead, use a Delegated Authentication Directory.
