cat atlassian-crowd.log |grep '2015-02-10 06:5' |grep 'scheduler_Worker-3'
2015-02-10 06:53:06,890 scheduler_Worker-3 INFO [atlassian.crowd.directory.DbCachingRemoteDirectory] INCREMENTAL synchronisation for directory [ 7274497 ] starting
2015-02-10 06:53:06,890 scheduler_Worker-3 INFO [atlassian.crowd.directory.DbCachingRemoteDirectory] Attempting INCREMENTAL synchronisation for directory [ 7274497 ]
2015-02-10 06:53:07,985 scheduler_Worker-3 INFO [crowd.directory.ldap.SpringLdapTemplateWrapper] Timed call for search with handler on dc=citrite,dc=net took 1018ms
2015-02-10 06:53:12,612 scheduler_Worker-3 INFO [directory.ldap.cache.UsnChangedCacheRefresher] scanned and compared [ 0 ] users to delete, [ 0 ] users to add, [ 192 ] users to update in DB cache in [ 5712ms ]
2015-02-10 06:53:12,618 scheduler_Worker-3 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] deleting [ 0 ] users
2015-02-10 06:53:12,620 scheduler_Worker-3 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] deleted [ 0 ] users in [ 2ms ]
2015-02-10 06:53:13,289 scheduler_Worker-3 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] scanning [ 192 ] users to add or update
2015-02-10 06:53:13,397 scheduler_Worker-3 INFO [atlassian.crowd.directory.DirectoryCacheImplUsingChangeOperations] scanned and compared [ 192 ] users for update in DB cache in [ 775ms ]
2015-02-10 06:53:13,397 scheduler_Worker-3 INFO [atlassian.crowd.directory.DirectoryCacheImplUsingChangeOperations] synchronised [ 192 ] users in [ 775ms ]
2015-02-10 06:53:14,098 scheduler_Worker-3 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] scanning [ 0 ] users to add or update
2015-02-10 06:53:14,211 scheduler_Worker-3 INFO [atlassian.crowd.directory.DirectoryCacheImplUsingChangeOperations] scanned and compared [ 0 ] users for update in DB cache in [ 809ms ]
2015-02-10 06:53:14,211 scheduler_Worker-3 INFO [atlassian.crowd.directory.DirectoryCacheImplUsingChangeOperations] synchronised [ 0 ] users in [ 809ms ]
2015-02-10 06:53:15,327 scheduler_Worker-3 INFO [crowd.directory.ldap.SpringLdapTemplateWrapper] Timed call for search with handler on dc=citrite,dc=net took 1115ms
2015-02-10 06:53:15,988 scheduler_Worker-3 INFO [directory.ldap.cache.UsnChangedCacheRefresher] found [ 3 ] changed remote groups in [ 1777ms ]
2015-02-10 06:53:15,988 scheduler_Worker-3 INFO [atlassian.crowd.directory.DirectoryCacheImplUsingChangeOperations] scanning [ 3 ] groups to add or update
2015-02-10 06:53:15,994 scheduler_Worker-3 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] scanned and compared [ 3 ] groups for update in DB cache in [ 6ms ]
2015-02-10 06:53:15,995 scheduler_Worker-3 INFO [atlassian.crowd.directory.DirectoryCacheImplUsingChangeOperations] synchronized [ 3 ] groups in [ 7ms ]
2015-02-10 06:53:18,220 scheduler_Worker-3 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] Could not remove user [ananthak] from group [confluence-users]. User was not found.
2015-02-10 06:53:20,317 scheduler_Worker-3 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] Could not remove user [shengl] from group [confluence-users]. User was not found.
2015-02-10 06:53:43,130 scheduler_Worker-3 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] removed [ 1686 ] user members from [ confluence-users ] in [ 26160ms ]
2015-02-10 06:53:45,611 scheduler_Worker-3 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] Could not remove user [ananthak] from group [jira-users]. User was not found.
2015-02-10 06:54:14,747 scheduler_Worker-3 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] removed [ 1854 ] user members from [ jira-users ] in [ 30661ms ]
2015-02-10 06:54:16,868 scheduler_Worker-3 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] Could not remove user [ananthak] from group [jira-internal]. User was not found.
2015-02-10 06:54:43,889 scheduler_Worker-3 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] removed [ 1791 ] user members from [ jira-internal ] in [ 28317ms ]
2015-02-10 06:54:43,898 scheduler_Worker-3 INFO [directory.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (3/3 - 100.0%) 87903ms elapsed
2015-02-10 06:54:43,899 scheduler_Worker-3 INFO [directory.ldap.cache.UsnChangedCacheRefresher] scanned and compared [ 0 ] groups for delete in DB cache in [ 0ms ]
2015-02-10 06:54:43,899 scheduler_Worker-3 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] removing [ 0 ] groups
2015-02-10 06:54:43,900 scheduler_Worker-3 INFO [atlassian.crowd.directory.DbCachingRemoteChangeOperations] removed [ 0 ] groups in [ 1ms ]
2015-02-10 06:54:43,900 scheduler_Worker-3 INFO [atlassian.crowd.directory.DbCachingRemoteDirectory] INCREMENTAL synchronisation complete for directory [ 7274497 ] in [ 97010ms ]
As a result of the extremely long synchronization times, the resource utilization of the machine will also shoot up.
We have multiple applications that synchronize their local caches with Crowd on 70-90 minute intervals.
After upgrading to 2.8.4, the CPU utilization shot up to 100%. We then realized that each crowd-to-app sync was taking over 1 hour to complete (previously took <5 min each).
Since the operations began to overlap and compete with one another, the load average shot up to 12+ on a 2 CPU machine.
After changing all of the directories back to the old default of 'aggregate group memberships' and waiting a few hours for all of the in-progress sync processes to complete, the load average went back to normal (below 1.0).
This is definitely a problem with the new default behavior of 'non-aggregating membership'. My advice, if you have a lot of directories or tons of people in your group memberships, stay away from the non-aggregating feature until it is better optimized.