Uploaded image for project: 'Jira Data Center'
  1. Jira Data Center
  2. JRASERVER-61759

JIRA performance degradation during directory sync

XMLWordPrintable

      Summary

      JIRA performance degrades significantly during full and incremental directory sync. CPU spiking to 100%, page load delay between .5 - 5 min.

      Environment

      • JIRA 7.0 or later
      • JIRA configured with AD LDAP directory (type Connected)

      Steps to Reproduce

      Steps to reproduce*

      1) Configure Microsoft AD in jira 7.1.4, make sure AD has enough users ie. 10k users
      2) Import few users 
      3) Login to jira as local user and be on directory page
      4) Configure jmeter "http" sessions using one of the AD user, session should repeat contentiously for 30 minute
      5) Initiate 20 concurrent sessions from jmeter --e.g browsing boards , issues etc.
      6) After 10 second of jemeter sessions initiate LDAP sync
      7) Login to jira on another browser and keep navigating pages, after 3-8 minute you will notice slowness and LDAP sync is running forever.

      Expected Results

      Directory sync should happen without having performance degradation.

      Actual Result

      JIRA performance significantly degrades when full and incremental sync is happening.

      Verification

      To verify if the instance is affected by this bug, collect thread dumps during slow performance as per Generate a Thread Dump - reviewing them the Caesium thread will contain the below thread over several thread dumps:

      "Caesium-1-3" #189 daemon prio=5 tid=0x00007f2eae212000 nid=0x3540 waiting on condition [0x00007f2f1cff9000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  <0x0000000543279b50> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:870)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1199)
	at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:943)
	at com.atlassian.cache.memory.DelegatingCache.removeAll(DelegatingCache.java:256)
	at com.atlassian.jira.application.DefaultApplicationRoleManager.clearUserCounts(DefaultApplicationRoleManager.java:632)
	at com.atlassian.jira.application.DefaultApplicationRoleManager.onUserDeleted(DefaultApplicationRoleManager.java:529)
	at sun.reflect.GeneratedMethodAccessor2429.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at com.atlassian.event.internal.SingleParameterMethodListenerInvoker.invoke(SingleParameterMethodListenerInvoker.java:36)
	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher$1$1.run(AsynchronousAbleEventDispatcher.java:48)
	at com.google.common.util.concurrent.MoreExecutors$DirectExecutorService.execute(MoreExecutors.java:299)
	at com.atlassian.event.internal.AsynchronousAbleEventDispatcher.dispatch(AsynchronousAbleEventDispatcher.java:107)
	at com.atlassian.event.internal.EventPublisherImpl.invokeListeners(EventPublisherImpl.java:160)
	at com.atlassian.event.internal.EventPublisherImpl.publish(EventPublisherImpl.java:79)
	at com.atlassian.crowd.directory.DbCachingRemoteChangeOperations.publishEvent(DbCachingRemoteChangeOperations.java:1062)
	at com.atlassian.crowd.directory.DbCachingRemoteChangeOperations.deleteCachedUsersByName(DbCachingRemoteChangeOperations.java:318)
	at com.atlassian.crowd.directory.DbCachingRemoteChangeOperations.deleteCachedUsersByGuid(DbCachingRemoteChangeOperations.java:285)
	at com.atlassian.crowd.directory.DirectoryCacheImplUsingChangeOperations.deleteCachedUsersByGuid(DirectoryCacheImplUsingChangeOperations.java:72)
	at com.atlassian.crowd.directory.ldap.cache.UsnChangedCacheRefresher.synchroniseUserChangesGuid(UsnChangedCacheRefresher.java:356)
	at com.atlassian.crowd.directory.ldap.cache.UsnChangedCacheRefresher.synchroniseUserChanges(UsnChangedCacheRefresher.java:381)
	at com.atlassian.crowd.directory.ldap.cache.UsnChangedCacheRefresher.synchroniseChanges(UsnChangedCacheRefresher.java:124)
	at com.atlassian.crowd.directory.DbCachingRemoteDirectory.synchroniseCache(DbCachingRemoteDirectory.java:1097)
	at com.atlassian.crowd.manager.directory.DirectorySynchroniserImpl.synchronise(DirectorySynchroniserImpl.java:76)
	at com.atlassian.jira.crowd.embedded.JiraDirectorySynchroniser.synchronizeDirectory(JiraDirectorySynchroniser.java:77)
	at com.atlassian.jira.crowd.embedded.JiraDirectorySynchroniser.runJob(JiraDirectorySynchroniser.java:52)
	at com.atlassian.scheduler.core.JobLauncher.runJob(JobLauncher.java:153)
	at com.atlassian.scheduler.core.JobLauncher.launchAndBuildResponse(JobLauncher.java:118)
	at com.atlassian.scheduler.core.JobLauncher.launch(JobLauncher.java:97)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.launchJob(CaesiumSchedulerService.java:401)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeClusteredJob(CaesiumSchedulerService.java:396)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService.executeQueuedJob(CaesiumSchedulerService.java:349)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:255)
	at com.atlassian.scheduler.caesium.impl.CaesiumSchedulerService$1.consume(CaesiumSchedulerService.java:252)
	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeJob(SchedulerQueueWorker.java:65)
	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.executeNextJob(SchedulerQueueWorker.java:59)
	at com.atlassian.scheduler.caesium.impl.SchedulerQueueWorker.run(SchedulerQueueWorker.java:34)
	at java.lang.Thread.run(Thread.java:745)

      And there will be a number of Tomcat worker threads waiting on futures inside that cache (as per the attached screenshot). Essentially the sync is invalidating a cache that a large number of actions rely on and when it's invalidated those threads will block until it's repopulated.

      Workaround 

      (If using Active Directory)

      1. Set the JIRA system property: 
        -Dcrowd.use.legacy.ad.incremental.sync=true

      This workaround will mean that users who no longer exist in AD but own content in JIRA will not be deleted from the cache on an INCREMENTAL sync only, thus not triggering this issue. A FULL sync will still be affected, however.

      Note on partial fix:

      We significantly reduced the performance problem by resolving issue -JRA-62742-, which was about skipping active user counting in cases when it was not required.

      After gathering feedback we decided to reopen this issue to look further into how can we improve user synchronisation performance. 

        1. sync.png
          sync.png
          81 kB
        2. Waiting on futures.jpg
          Waiting on futures.jpg
          627 kB

            [JRASERVER-61759] JIRA performance degradation during directory sync

            Adam Jakubowski (Inactive) added a comment - - edited

            We made significant improvements to license count recalculation in JIRA 7.2.9. This means that problem, described in this ticket, will be resolved in most cases with upgrade to 7.2.9. At the moment we don't have tests to quantify how 7.2.9 will behave compared to 7.3.3 during directory sync. This is why this ticket is not marked as solved in 7.2.9 as this problem may sometimes appear under heavy load and during long sync in 7.2.9. 

            If possible we strongly recommend upgrading to 7.3.3+ to resolve this problem. 

            Adam Jakubowski (Inactive) added a comment - - edited We made significant improvements to license count recalculation in JIRA 7.2.9. This means that problem, described in this ticket, will be resolved in most cases with upgrade to 7.2.9. At the moment we don't have tests to quantify how 7.2.9 will behave compared to 7.3.3 during directory sync. This is why this ticket is not marked as solved in 7.2.9 as this problem may sometimes appear under heavy load and during long sync in 7.2.9.  If possible we strongly recommend upgrading to 7.3.3+ to resolve this problem. 

            Matias Massone added a comment -

            Hello izinoviev,

            Thanks a lot for your answer. My only concern is that we have unlimited users license, as far as I saw on the ticket you shared, it should be not affecting that kind of license. I'll do a further investigation and I will let you know if I find something else.

             

            Best Regards

             

            Matias Massone added a comment - Hello  izinoviev , Thanks a lot for your answer. My only concern is that we have unlimited users license, as far as I saw on the ticket you shared, it should be not affecting that kind of license. I'll do a further investigation and I will let you know if I find something else.   Best Regards  

            Ilya Zinoviev (Inactive) added a comment - - edited

            Hi matiasx.massone1782953930,

            Unfortunately there is no way to change the way of license recalculation. It was fixed deep inside on JIRA code.

            The only way you can get it is to update to 7.3.3+.

            If you are not going to update to 7.3 in near future, there is a partial fix of this issue for 7.2.9(which is going to be released soon): https://jira.atlassian.com/browse/JRASERVER-64384

            Ilya Zinoviev
            JIRA development

            Ilya Zinoviev (Inactive) added a comment - - edited Hi matiasx.massone1782953930 , Unfortunately there is no way to change the way of license recalculation. It was fixed deep inside on JIRA code. The only way you can get it is to update to 7.3.3+. If you are not going to update to 7.3 in near future, there is a partial fix of this issue for 7.2.9(which is going to be released soon): https://jira.atlassian.com/browse/JRASERVER-64384 Ilya Zinoviev JIRA development

            Matias Massone added a comment -

            Hello Ilya Zinoviev,

             

            How did you do that change? I have the same problem on one of our test servers and I couldn't find where to change the license recalc

             

            Kind Regards

             

            Matias Massone added a comment - Hello Ilya Zinoviev,   How did you do that change? I have the same problem on one of our test servers and I couldn't find where to change the license recalc   Kind Regards  

            Ilya Zinoviev (Inactive) added a comment - - edited

            While investigating this issue we found out that the problem was caused by JIRA recalculating license information for each synchronized user, group and user-group membership.
            When JIRA is recalculating this information all user threads are blocked waiting for this. Such recalculation may even happen couple of times during one customer request.

            The fix was to recalculate this license only once: when synchronization is finished(or fails)

            Numbers:
            Current results:
            JMeter configuration: 7 threads, 200msec wait between requests
            LDAP: 60k users, 50k groups.

            Setup Sync time, s Response time, ms/req
            Before fix 7670 ~1000
            After fix 4836 370

            Ilya Zinoviev
            JIRA development

            Ilya Zinoviev (Inactive) added a comment - - edited While investigating this issue we found out that the problem was caused by JIRA recalculating license information for each synchronized user, group and user-group membership. When JIRA is recalculating this information all user threads are blocked waiting for this. Such recalculation may even happen couple of times during one customer request. The fix was to recalculate this license only once: when synchronization is finished(or fails) Numbers: Current results: JMeter configuration: 7 threads, 200msec wait between requests LDAP: 60k users, 50k groups. Setup Sync time, s Response time, ms/req Before fix 7670 ~1000 After fix 4836 370 Ilya Zinoviev JIRA development

            gzgenm added a comment -

            JIRA 6.4. We have 14k users in one group in OpenLDAP. A sync process is going increasingly slow until it ends up being stuck which makes JIRA extremely slow.

            gzgenm added a comment - JIRA 6.4. We have 14k users in one group in OpenLDAP. A sync process is going increasingly slow until it ends up being stuck which makes JIRA extremely slow.

            Przemyslaw Czuj (Inactive) added a comment -

            michael.ferry, djbdjb00djb

            We reopened this issue, as the fix for JRA-62742 did not completely resolve user directory synchronisation performance issue for all customers.
            We are investigating the problem and we will inform you about status of resolution within this issue.

            For now we can see that the actual problem has most likely multiple causes and we fixed only one of them. 

            Cheers,
            Przemysław Czuj [JIRA Developer]

            Przemyslaw Czuj (Inactive) added a comment - michael.ferry , djbdjb00djb We reopened this issue, as the fix for JRA-62742  did not completely resolve user directory synchronisation performance issue for all customers. We are investigating the problem and we will inform you about status of resolution within this issue. For now we can see that the actual problem has most likely multiple causes and we fixed only one of them.  Cheers, Przemysław Czuj [JIRA Developer]

            Michael Ferry added a comment -

            We are experiencing this issue in JIRA 7.2.3, with a "Generic Directory Server" LDAP configuration.  

            JDK 1.8.0_102 on Oracle Linux 6 (like CentOS, RedHat)

            The System Property above does not help.  

            More info about our case and configuration is in Support Request GHS-62159

            Michael Ferry added a comment - We are experiencing this issue in JIRA 7.2.3, with a "Generic Directory Server" LDAP configuration.   JDK 1.8.0_102 on Oracle Linux 6 (like CentOS, RedHat) The System Property above does not help.   More info about our case and configuration is in Support Request GHS-62159

            Jabari Deng added a comment -

            JIRA performance significantly degrades when [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group

            Jabari Deng added a comment - JIRA performance significantly degrades when [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group

            Jabari Deng added a comment - 
            2016-11-02 12:44:35,990 Caesium-1-2 INFO ServiceRunner     [c.a.crowd.directory.DirectoryCacheImplUsingChangeOperations] synchronised [ 59522 ] users in [ 2293ms ]
2016-11-02 12:45:06,332 Caesium-1-2 INFO ServiceRunner     [c.a.crowd.directory.DirectoryCacheImplUsingChangeOperations] scanning [ 14257 ] groups to add or update
2016-11-02 12:45:07,198 Caesium-1-2 INFO ServiceRunner     [c.a.crowd.directory.DirectoryCacheImplUsingChangeOperations] synchronized [ 14257 ] groups in [ 866ms ]
2016-11-02 12:45:12,535 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] finished group attribute sync with 0 failures in [ 5337ms ]
2016-11-02 12:46:19,756 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (673/14257 - 04.7%) 67195ms elapsed
2016-11-02 12:47:41,500 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (768/14257 - 05.4%) 148942ms elapsed
2016-11-02 12:48:41,503 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (1343/14257 - 09.4%) 208945ms elapsed
2016-11-02 12:49:54,412 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (1728/14257 - 12.1%) 281854ms elapsed
2016-11-02 12:53:49,687 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (2617/14257 - 18.4%) 517129ms elapsed
2016-11-02 12:55:04,691 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (3336/14257 - 23.4%) 592132ms elapsed
2016-11-02 12:56:21,240 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (3873/14257 - 27.2%) 668681ms elapsed
2016-11-02 12:57:24,515 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (4004/14257 - 28.1%) 731957ms elapsed
2016-11-02 12:58:43,837 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (4494/14257 - 31.5%) 811279ms elapsed
2016-11-02 12:59:43,838 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (4954/14257 - 34.7%) 871280ms elapsed
2016-11-02 13:00:59,096 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (5936/14257 - 41.6%) 946538ms elapsed
2016-11-02 13:02:06,125 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (6900/14257 - 48.4%) 1013567ms elapsed
2016-11-02 13:03:07,439 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (7522/14257 - 52.8%) 1074881ms elapsed
2016-11-02 13:04:16,280 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (8672/14257 - 60.8%) 1143722ms elapsed
2016-11-02 13:05:27,370 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (10265/14257 - 72.0%) 1214812ms elapsed
2016-11-02 13:06:30,150 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (11199/14257 - 78.6%) 1277592ms elapsed
2016-11-02 13:07:35,982 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (11916/14257 - 83.6%) 1343424ms elapsed
2016-11-02 13:08:42,192 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (12494/14257 - 87.6%) 1409634ms elapsed
2016-11-02 13:09:48,650 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (13038/14257 - 91.4%) 1476092ms elapsed
2016-11-02 13:10:49,696 Caesium-1-2 INFO ServiceRunner     [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (14193/14257 - 99.6%) 1537137ms elapsed

            Jabari Deng added a comment - 2016-11-02 12:44:35,990 Caesium-1-2 INFO ServiceRunner [c.a.crowd.directory.DirectoryCacheImplUsingChangeOperations] synchronised [ 59522 ] users in [ 2293ms ] 2016-11-02 12:45:06,332 Caesium-1-2 INFO ServiceRunner [c.a.crowd.directory.DirectoryCacheImplUsingChangeOperations] scanning [ 14257 ] groups to add or update 2016-11-02 12:45:07,198 Caesium-1-2 INFO ServiceRunner [c.a.crowd.directory.DirectoryCacheImplUsingChangeOperations] synchronized [ 14257 ] groups in [ 866ms ] 2016-11-02 12:45:12,535 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] finished group attribute sync with 0 failures in [ 5337ms ] 2016-11-02 12:46:19,756 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (673/14257 - 04.7%) 67195ms elapsed 2016-11-02 12:47:41,500 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (768/14257 - 05.4%) 148942ms elapsed 2016-11-02 12:48:41,503 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (1343/14257 - 09.4%) 208945ms elapsed 2016-11-02 12:49:54,412 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (1728/14257 - 12.1%) 281854ms elapsed 2016-11-02 12:53:49,687 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (2617/14257 - 18.4%) 517129ms elapsed 2016-11-02 12:55:04,691 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (3336/14257 - 23.4%) 592132ms elapsed 2016-11-02 12:56:21,240 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (3873/14257 - 27.2%) 668681ms elapsed 2016-11-02 12:57:24,515 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (4004/14257 - 28.1%) 731957ms elapsed 2016-11-02 12:58:43,837 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (4494/14257 - 31.5%) 811279ms elapsed 2016-11-02 12:59:43,838 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (4954/14257 - 34.7%) 871280ms elapsed 2016-11-02 13:00:59,096 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (5936/14257 - 41.6%) 946538ms elapsed 2016-11-02 13:02:06,125 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (6900/14257 - 48.4%) 1013567ms elapsed 2016-11-02 13:03:07,439 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (7522/14257 - 52.8%) 1074881ms elapsed 2016-11-02 13:04:16,280 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (8672/14257 - 60.8%) 1143722ms elapsed 2016-11-02 13:05:27,370 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (10265/14257 - 72.0%) 1214812ms elapsed 2016-11-02 13:06:30,150 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (11199/14257 - 78.6%) 1277592ms elapsed 2016-11-02 13:07:35,982 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (11916/14257 - 83.6%) 1343424ms elapsed 2016-11-02 13:08:42,192 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (12494/14257 - 87.6%) 1409634ms elapsed 2016-11-02 13:09:48,650 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (13038/14257 - 91.4%) 1476092ms elapsed 2016-11-02 13:10:49,696 Caesium-1-2 INFO ServiceRunner [c.a.c.d.ldap.cache.AbstractCacheRefresher] migrated memberships for group - (14193/14257 - 99.6%) 1537137ms elapsed

              izinoviev Ilya Zinoviev (Inactive)
              vkharisma vkharisma
              Affected customers:
              19 This affects my team
              Watchers:
              50 Start watching this issue

                Created:
                Updated:
                Resolved: