-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Medium
-
Affects Version/s: 4.2.5, 4.3.7, 4.4.1, 5.0.0
-
Component/s: Core features, Performance
-
None
-
3
-
Severity 2 - Major
-
0
Issue Summary
The performance of a directory synchronization degrades due to long garbage collection pauses which eventually lead to out of memory errors when memberships are being re-built for a large number of groups, with a large number of members each and if many of those members are found across the different groups.
The case observed had approximately 50 groups, where 10 out of those 50 had 60000 members each where and a large number of those members appeared across those 10 groups when the heap dump was taken.
Steps to Reproduce
There are not specific steps to reproduce this issue and as mentioned above it is a very specific case that could lead to this problem.
Expected Results
The directory synchronization should not cause large memory allocations for this case.
Actual Results
The below outputs were taken from a heap dump generated while the synchronization was running. The details provided below are meant to help diagnosing this issue when a heap dump is analyzed as there will be:
- a large memory allocation by a thread (in the output below Caesium thread) while performs the directory synchronization (see below "Dominator Tree").
- RFC4519DirectoryMembershipsIterable$MembershipHolder as part of the memory allocation by the Caesium thread.
- an ImmutableSet with a large number of javax.naming.ldap.LdapName (in the output below approximately 65000) which represent the members of a group.
Dominator Tree - Top-4 Dominators
Class Name | Shallow Heap | Retained Heap | Percentage ------------------------------------------------------------------------------------------------------------------------------------------ java.lang.Thread @ 0x69e86b2e0 Caesium-1-3 Thread | 0.00 MB | 4,422.02 MB | 80.54% com.hazelcast.hibernate.local.LocalRegionCache @ 0x68a276010 | 0.00 MB | 62.11 MB | 1.13% java.util.WeakHashMap$Entry @ 0x6cb445e28 | 0.00 MB | 26.25 MB | 0.48% com.atlassian.confluence.plugins.gatekeeper.evaluator.cache.EvaluatorCacheHolder @ 0x696fb49a8| 0.00 MB | 24.64 MB | 0.45% ------------------------------------------------------------------------------------------------------------------------------------------
Ceasium Thread
Class Name | Shallow Heap | Retained Heap | Percentage ------------------------------------------------------------------------------------------------------------------------------------------------------------- java.lang.Thread @ 0x69e86b2e0 Caesium-1-3 Thread | 0.00 MB | 4,422.02 MB | 80.54% |- java.util.ArrayList @ 0x6f4100070 | 0.00 MB | 3,583.91 MB | 65.27% | '- java.lang.Object[73] @ 0x7983ba9a0 | 0.00 MB | 3,583.91 MB | 65.27% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7982000d0| 0.00 MB | 126.45 MB | 2.30% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7bfe000b0| 0.00 MB | 124.34 MB | 2.26% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x6b8a06688| 0.00 MB | 123.74 MB | 2.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7b03dbbe0| 0.00 MB | 123.74 MB | 2.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x706b50770| 0.00 MB | 123.74 MB | 2.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x700255810| 0.00 MB | 123.74 MB | 2.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x75e600020| 0.00 MB | 123.74 MB | 2.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7ea198008| 0.00 MB | 123.74 MB | 2.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x77ef00060| 0.00 MB | 123.74 MB | 2.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x74f600110| 0.00 MB | 123.74 MB | 2.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7c8acf6e0| 0.00 MB | 123.72 MB | 2.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7b6f000f0| 0.00 MB | 123.69 MB | 2.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x78cb59350| 0.00 MB | 123.69 MB | 2.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7e959ab70| 0.00 MB | 114.14 MB | 2.08% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x77932aff8| 0.00 MB | 88.27 MB | 1.61% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7d4403cf8| 0.00 MB | 75.10 MB | 1.37% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x6ff662120| 0.00 MB | 69.28 MB | 1.26% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x795264dd0| 0.00 MB | 69.01 MB | 1.26% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7a55000d0| 0.00 MB | 68.79 MB | 1.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7e959c988| 0.00 MB | 68.66 MB | 1.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x6fb1d7130| 0.00 MB | 68.59 MB | 1.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x71fb5dab0| 0.00 MB | 68.57 MB | 1.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x735a000b0| 0.00 MB | 68.57 MB | 1.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x71f666018| 0.00 MB | 68.57 MB | 1.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x6f4ab8da8| 0.00 MB | 68.57 MB | 1.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x707fbf200| 0.00 MB | 68.57 MB | 1.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x740b1cf18| 0.00 MB | 68.57 MB | 1.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x680f33b40| 0.00 MB | 68.57 MB | 1.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x6eec69d00| 0.00 MB | 59.91 MB | 1.09% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x69e6467e8| 0.00 MB | 59.12 MB | 1.08% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7884dcd50| 0.00 MB | 46.85 MB | 0.85% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7577ffc68| 0.00 MB | 46.34 MB | 0.84% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7ab1ffbb8| 0.00 MB | 44.97 MB | 0.82% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7a3f9d7c8| 0.00 MB | 44.95 MB | 0.82% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x72c966810| 0.00 MB | 43.89 MB | 0.80% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x6b2337eb8| 0.00 MB | 43.83 MB | 0.80% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7502ad050| 0.00 MB | 43.49 MB | 0.79% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x75e8e19e8| 0.00 MB | 35.83 MB | 0.65% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x71f6000b0| 0.00 MB | 34.44 MB | 0.63% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x73111b730| 0.00 MB | 33.62 MB | 0.61% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x78918e158| 0.00 MB | 29.21 MB | 0.53% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7ac800020| 0.00 MB | 29.19 MB | 0.53% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x706f99830| 0.00 MB | 27.51 MB | 0.50% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7c0549130| 0.00 MB | 26.94 MB | 0.49% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x72ca9e020| 0.00 MB | 26.91 MB | 0.49% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x757538c30| 0.00 MB | 18.42 MB | 0.34% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7d9200de8| 0.00 MB | 16.51 MB | 0.30% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7d4bce6e8| 0.00 MB | 14.94 MB | 0.27% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7983baad8| 0.00 MB | 13.82 MB | 0.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x77926b720| 0.00 MB | 13.74 MB | 0.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7a5500210| 0.00 MB | 13.69 MB | 0.25% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x75770b030| 0.00 MB | 12.80 MB | 0.23% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7879032b0| 0.00 MB | 12.48 MB | 0.23% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x730e73028| 0.00 MB | 12.29 MB | 0.22% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x76c472000| 0.00 MB | 12.02 MB | 0.22% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x707cd1290| 0.00 MB | 11.25 MB | 0.20% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x75eae1170| 0.00 MB | 11.23 MB | 0.20% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7366107d8| 0.00 MB | 9.90 MB | 0.18% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x77accdbd0| 0.00 MB | 7.23 MB | 0.13% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x6ff701c78| 0.00 MB | 5.98 MB | 0.11% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x69ac025e8| 0.00 MB | 4.25 MB | 0.08% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7d9500090| 0.00 MB | 0.81 MB | 0.01% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7e385eb88| 0.00 MB | 0.74 MB | 0.01% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7ef400308| 0.00 MB | 0.42 MB | 0.01% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7d4400130| 0.00 MB | 0.41 MB | 0.01% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x72a758178| 0.00 MB | 0.34 MB | 0.01% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7e959ab88| 0.00 MB | 0.01 MB | 0.00% | |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x798a7e9e8| 0.00 MB | 0.01 MB | 0.00% | '- Total: 68 entries | | | -------------------------------------------------------------------------------------------------------------------------------------------------------------
MembershipHolder
Class Name | Shallow Heap | Retained Heap | Percentage ---------------------------------------------------------------------------------------------------------------------------------------------------- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7982000d0| 0.00 MB | 126.45 MB | 2.30% '- com.google.common.collect.RegularImmutableSet @ 0x756aa2d90 | 0.00 MB | 126.45 MB | 2.30% |- java.lang.Object[131072] @ 0x7a3300000 | 0.50 MB | 0.50 MB | 0.01% |- java.lang.Object[68223] @ 0x756aa2db0 | 0.26 MB | 0.26 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x7a2deec48 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x7a3103328 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x7a3189f08 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x7b13f4c28 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x77eab4548 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x77eabbcc0 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x77eafa038 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x7a2de0578 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x7a3154218 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x7b10c3c88 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x7b1358cb8 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x756af2108 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x756b13370 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x756b17b78 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x756b48a80 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x756b828c8 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x756b97ee0 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x77ea58f48 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x77eab8c40 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x77eac3120 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x77eade408 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x77eae2cc8 | 0.00 MB | 0.00 MB | 0.00% |- javax.naming.ldap.LdapName @ 0x77eae5780 | 0.00 MB | 0.00 MB | 0.00% '- Total: 25 of 68,225 entries; 68,200 more | | | ----------------------------------------------------------------------------------------------------------------------------------------------------
Workaround
Currently there is no known workaround for this behavior. A workaround will be added here when available
- is incorporated by
-
-
- Closed
-
- relates to
-
KRAK-4604 Loading...