Deleting a group during synchronisation leads to error 32 and synchronisation will not complete

This was reported by a customer with a very large number of users in there LDAP directory (300K+), running OpenLDAP with Confluence. The problem is that synchronisation takes many hours, and groups are constantly being created deleted etc in the directory. If a group is deleted while the memberships are being synchronised then an exception will be thrown and the synchronisation will not complete. Here is the error:

2014-10-08 17:59:53,698 ERROR [scheduler_Worker-9] [atlassian.crowd.directory.DbCachingDirectoryPoller] pollChanges Error occurred while refreshing the cache for directory [ 904953859 ].
org.springframework.ldap.NameNotFoundException: [LDAP: error code 32 - No Such Object]; nested exception is javax.naming.NameNotFoundException: [LDAP: error code 32 - No Such Object]; blah name 'cn=BATCH ABEND REPORT,ou=Groups,o=blah.com'
at org.springframework.ldap.support.LdapUtils.convertLdapException(LdapUtils.java:174)
at org.springframework.ldap.core.LdapTemplate.executeWithContext(LdapTemplate.java:810)
at org.springframework.ldap.core.LdapTemplate.executeReadOnly(LdapTemplate.java:793)
at org.springframework.ldap.core.LdapTemplate.lookup(LdapTemplate.java:935)
at com.atlassian.crowd.directory.ldap.LdapTemplateWithClassLoaderWrapper$9.call(LdapTemplateWithClassLoaderWrapper.java:159)
at com.atlassian.crowd.directory.ldap.LdapTemplateWithClassLoaderWrapper.invokeWithContextClassLoader(LdapTemplateWithClassLoaderWrapper.java:54)
at com.atlassian.crowd.directory.ldap.LdapTemplateWithClassLoaderWrapper.lookup(LdapTemplateWithClassLoaderWrapper.java:155)
at com.atlassian.crowd.directory.RFC4519Directory.findDirectMembersOfGroup(RFC4519Directory.java:938)
at com.atlassian.crowd.directory.RFC4519DirectoryMembershipsIterable$2.apply(RFC4519DirectoryMembershipsIterable.java:77)
at com.atlassian.crowd.directory.RFC4519DirectoryMembershipsIterable$2.apply(RFC4519DirectoryMembershipsIterable.java:69)
at com.google.common.collect.Iterators$8.next(Iterators.java:782)
at com.atlassian.crowd.directory.ldap.cache.AbstractCacheRefresher.synchroniseMemberships(AbstractCacheRefresher.java:126)
at com.atlassian.crowd.directory.ldap.cache.AbstractCacheRefresher.synchroniseAll(AbstractCacheRefresher.java:82)
at com.atlassian.crowd.directory.DbCachingRemoteDirectory.synchroniseCache(DbCachingRemoteDirectory.java:1008)
at com.atlassian.crowd.manager.directory.DirectorySynchroniserImpl.synchronise(DirectorySynchroniserImpl.java:75)
at com.atlassian.crowd.directory.DbCachingDirectoryPoller.pollChanges(DbCachingDirectoryPoller.java:50)
at com.atlassian.crowd.manager.directory.monitor.poller.DirectoryPollerJobBean.executeInternal(DirectoryPollerJobBean.java:29)
at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
at org.quartz.core.JobRunShell.run(JobRunShell.java:199)
at com.atlassian.confluence.schedule.quartz.ConfluenceQuartzThreadPool$1.run(ConfluenceQuartzThreadPool.java:20)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Caused by: javax.naming.NameNotFoundException: [LDAP: error code 32 - No Such Object]; r906680674ining name 'cn=BATCH ABEND REPORT,ou=Groups,o=hp.com'
at com.sun.jndi.ldap.LdapCtx.mapErrorCode(LdapCtx.java:3112)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:3033)
at com.sun.jndi.ldap.LdapCtx.processReturnCode(LdapCtx.java:2840)
at com.sun.jndi.ldap.LdapCtx.c_getAttributes(LdapCtx.java:1332)
at com.sun.jndi.toolkit.ctx.ComponentDirContext.p_getAttributes(ComponentDirContext.java:231)
at com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.getAttributes(PartialCompositeDirContext.java:139)
at javax.naming.directory.InitialDirContext.getAttributes(InitialDirContext.java:152)
at sun.reflect.GeneratedMethodAccessor773.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.ldap.transaction.compensating.manager.TransactionAwareDirContextInvocationHandler.invoke(TransactionAwareDirContextInvocationHandler.java:92)
at com.sun.proxy.$Proxy2058.getAttributes(Unknown Source)
at org.springframework.ldap.core.LdapTemplate$17.executeWithContext(LdapTemplate.java:937)
at org.springframework.ldap.core.LdapTemplate.executeWithContext(LdapTemplate.java:807)
... 19 more

I was able to reproduce what I think is the same problem with ApacheDS and some help from the debugger. Reproduction steps:

1. Set up ApacheDS with some users and groups
2. In Confluence configure the user directory
3. Attach the debugger and set a breakpoint in AbstractCacheRefresher#synchroniseMemberships. I put the break point on like 121, after the call to getMemberships(groupsByName.keySet());, but before the code has started iterating through the memberships collection.
4. In confluence synchronise the directory
5. Wait for your breakpoint to be hit
6. Now delete one of the groups form your LDAP directory
7. Now allow the process to continue
8. Observe that the synchronisation fails with LDAP error code 32.