[CWD-5803] RFC4519DirectoryMembershipsIterable causes larger memory allocations during directory synchronization

Type: Bug
Resolution: Fixed
Priority: Medium
Fix Version/s: 4.3.8, 4.4.2, 5.0.1
Affects Version/s: 4.2.5, 4.3.7, 4.4.1, 5.0.0
Component/s: Core features, Performance
Labels:
None

Support reference count:
3
Symptom Severity:
Severity 2 - Major
UIS:
0
Bug Fix Policy:
View Atlassian Server bug fix policy

Issue Summary

The performance of a directory synchronization degrades due to long garbage collection pauses which eventually lead to out of memory errors when memberships are being re-built for a large number of groups, with a large number of members each and if many of those members are found across the different groups.

The case observed had approximately 50 groups, where 10 out of those 50 had 60000 members each where and a large number of those members appeared across those 10 groups when the heap dump was taken.

Steps to Reproduce

There are not specific steps to reproduce this issue and as mentioned above it is a very specific case that could lead to this problem.

Expected Results

The directory synchronization should not cause large memory allocations for this case.

Actual Results

The below outputs were taken from a heap dump generated while the synchronization was running. The details provided below are meant to help diagnosing this issue when a heap dump is analyzed as there will be:

a large memory allocation by a thread (in the output below Caesium thread) while performs the directory synchronization (see below "Dominator Tree").
RFC4519DirectoryMembershipsIterable$MembershipHolder as part of the memory allocation by the Caesium thread.
an ImmutableSet with a large number of javax.naming.ldap.LdapName (in the output below approximately 65000) which represent the members of a group.

Dominator Tree - Top-4 Dominators

Class Name                                                                                    | Shallow Heap | Retained Heap | Percentage
------------------------------------------------------------------------------------------------------------------------------------------
java.lang.Thread @ 0x69e86b2e0  Caesium-1-3 Thread                                            |      0.00 MB |   4,422.02 MB |     80.54%
com.hazelcast.hibernate.local.LocalRegionCache @ 0x68a276010                                  |      0.00 MB |      62.11 MB |      1.13%
java.util.WeakHashMap$Entry @ 0x6cb445e28                                                     |      0.00 MB |      26.25 MB |      0.48%
com.atlassian.confluence.plugins.gatekeeper.evaluator.cache.EvaluatorCacheHolder @ 0x696fb49a8|      0.00 MB |      24.64 MB |      0.45%
------------------------------------------------------------------------------------------------------------------------------------------

Ceasium Thread

Class Name                                                                                                       | Shallow Heap | Retained Heap | Percentage
-------------------------------------------------------------------------------------------------------------------------------------------------------------
java.lang.Thread @ 0x69e86b2e0  Caesium-1-3 Thread                                                               |      0.00 MB |   4,422.02 MB |     80.54%
|- java.util.ArrayList @ 0x6f4100070                                                                             |      0.00 MB |   3,583.91 MB |     65.27%
|  '- java.lang.Object[73] @ 0x7983ba9a0                                                                         |      0.00 MB |   3,583.91 MB |     65.27%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7982000d0|      0.00 MB |     126.45 MB |      2.30%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7bfe000b0|      0.00 MB |     124.34 MB |      2.26%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x6b8a06688|      0.00 MB |     123.74 MB |      2.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7b03dbbe0|      0.00 MB |     123.74 MB |      2.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x706b50770|      0.00 MB |     123.74 MB |      2.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x700255810|      0.00 MB |     123.74 MB |      2.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x75e600020|      0.00 MB |     123.74 MB |      2.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7ea198008|      0.00 MB |     123.74 MB |      2.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x77ef00060|      0.00 MB |     123.74 MB |      2.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x74f600110|      0.00 MB |     123.74 MB |      2.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7c8acf6e0|      0.00 MB |     123.72 MB |      2.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7b6f000f0|      0.00 MB |     123.69 MB |      2.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x78cb59350|      0.00 MB |     123.69 MB |      2.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7e959ab70|      0.00 MB |     114.14 MB |      2.08%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x77932aff8|      0.00 MB |      88.27 MB |      1.61%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7d4403cf8|      0.00 MB |      75.10 MB |      1.37%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x6ff662120|      0.00 MB |      69.28 MB |      1.26%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x795264dd0|      0.00 MB |      69.01 MB |      1.26%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7a55000d0|      0.00 MB |      68.79 MB |      1.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7e959c988|      0.00 MB |      68.66 MB |      1.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x6fb1d7130|      0.00 MB |      68.59 MB |      1.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x71fb5dab0|      0.00 MB |      68.57 MB |      1.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x735a000b0|      0.00 MB |      68.57 MB |      1.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x71f666018|      0.00 MB |      68.57 MB |      1.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x6f4ab8da8|      0.00 MB |      68.57 MB |      1.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x707fbf200|      0.00 MB |      68.57 MB |      1.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x740b1cf18|      0.00 MB |      68.57 MB |      1.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x680f33b40|      0.00 MB |      68.57 MB |      1.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x6eec69d00|      0.00 MB |      59.91 MB |      1.09%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x69e6467e8|      0.00 MB |      59.12 MB |      1.08%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7884dcd50|      0.00 MB |      46.85 MB |      0.85%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7577ffc68|      0.00 MB |      46.34 MB |      0.84%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7ab1ffbb8|      0.00 MB |      44.97 MB |      0.82%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7a3f9d7c8|      0.00 MB |      44.95 MB |      0.82%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x72c966810|      0.00 MB |      43.89 MB |      0.80%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x6b2337eb8|      0.00 MB |      43.83 MB |      0.80%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7502ad050|      0.00 MB |      43.49 MB |      0.79%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x75e8e19e8|      0.00 MB |      35.83 MB |      0.65%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x71f6000b0|      0.00 MB |      34.44 MB |      0.63%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x73111b730|      0.00 MB |      33.62 MB |      0.61%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x78918e158|      0.00 MB |      29.21 MB |      0.53%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7ac800020|      0.00 MB |      29.19 MB |      0.53%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x706f99830|      0.00 MB |      27.51 MB |      0.50%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7c0549130|      0.00 MB |      26.94 MB |      0.49%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x72ca9e020|      0.00 MB |      26.91 MB |      0.49%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x757538c30|      0.00 MB |      18.42 MB |      0.34%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7d9200de8|      0.00 MB |      16.51 MB |      0.30%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7d4bce6e8|      0.00 MB |      14.94 MB |      0.27%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7983baad8|      0.00 MB |      13.82 MB |      0.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x77926b720|      0.00 MB |      13.74 MB |      0.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7a5500210|      0.00 MB |      13.69 MB |      0.25%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x75770b030|      0.00 MB |      12.80 MB |      0.23%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7879032b0|      0.00 MB |      12.48 MB |      0.23%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x730e73028|      0.00 MB |      12.29 MB |      0.22%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x76c472000|      0.00 MB |      12.02 MB |      0.22%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x707cd1290|      0.00 MB |      11.25 MB |      0.20%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x75eae1170|      0.00 MB |      11.23 MB |      0.20%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7366107d8|      0.00 MB |       9.90 MB |      0.18%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x77accdbd0|      0.00 MB |       7.23 MB |      0.13%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x6ff701c78|      0.00 MB |       5.98 MB |      0.11%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x69ac025e8|      0.00 MB |       4.25 MB |      0.08%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7d9500090|      0.00 MB |       0.81 MB |      0.01%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7e385eb88|      0.00 MB |       0.74 MB |      0.01%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7ef400308|      0.00 MB |       0.42 MB |      0.01%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7d4400130|      0.00 MB |       0.41 MB |      0.01%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x72a758178|      0.00 MB |       0.34 MB |      0.01%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7e959ab88|      0.00 MB |       0.01 MB |      0.00%
|     |- com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x798a7e9e8|      0.00 MB |       0.01 MB |      0.00%
|     '- Total: 68 entries                                                                                       |              |               |           
-------------------------------------------------------------------------------------------------------------------------------------------------------------

MembershipHolder

Class Name                                                                                              | Shallow Heap | Retained Heap | Percentage
----------------------------------------------------------------------------------------------------------------------------------------------------
com.atlassian.crowd.directory.rfc4519.RFC4519DirectoryMembershipsIterable$MembershipHolder @ 0x7982000d0|      0.00 MB |     126.45 MB |      2.30%
'- com.google.common.collect.RegularImmutableSet @ 0x756aa2d90                                          |      0.00 MB |     126.45 MB |      2.30%
   |- java.lang.Object[131072] @ 0x7a3300000                                                            |      0.50 MB |       0.50 MB |      0.01%
   |- java.lang.Object[68223] @ 0x756aa2db0                                                             |      0.26 MB |       0.26 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x7a2deec48                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x7a3103328                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x7a3189f08                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x7b13f4c28                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x77eab4548                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x77eabbcc0                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x77eafa038                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x7a2de0578                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x7a3154218                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x7b10c3c88                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x7b1358cb8                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x756af2108                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x756b13370                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x756b17b78                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x756b48a80                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x756b828c8                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x756b97ee0                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x77ea58f48                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x77eab8c40                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x77eac3120                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x77eade408                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x77eae2cc8                                                          |      0.00 MB |       0.00 MB |      0.00%
   |- javax.naming.ldap.LdapName @ 0x77eae5780                                                          |      0.00 MB |       0.00 MB |      0.00%
   '- Total: 25 of 68,225 entries; 68,200 more                                                          |              |               |           
----------------------------------------------------------------------------------------------------------------------------------------------------

Workaround

Currently there is no known workaround for this behavior. A workaround will be added here when available

is incorporated by

CONFSERVER-81017 Upgrade Confluence Embedded Crowd Mitigate CWD-5803

Closed

mentioned in: Page Failed to load; Page Failed to load; Page Failed to load

relates to: KRAK-4604 You do not have permission to view this issue

aanastasov made changes - 25/Oct/2023 1:22 PM

Remote Link

Original: This issue links to "KRAK-4604 (Bulldog)" [ 659420 ]

New: This issue links to "KRAK-4604 (JIRA Server (Bulldog))" [ 659420 ]

Yingran Sun made changes - 10/Mar/2023 6:16 AM

Remote Link

New: This issue links to "Page (Confluence)" [ 739457 ]

Rudy Slaiby made changes - 02/Mar/2023 8:45 PM

Remote Link

New: This issue links to "Page (Confluence)" [ 736654 ]

Themis made changes - 23/Feb/2023 3:03 PM

Remote Link

New: This issue links to "Page (Confluence)" [ 734397 ]

Richard Atkins made changes - 16/Nov/2022 10:39 PM

Link

New: This issue is incorporated by ~~CONFSERVER-81017~~ [ ~~CONFSERVER-81017~~ ]

Esteban Casuscelli made changes - 05/Jul/2022 4:06 PM

Resolution		New: Fixed [ 1 ]
Status	Original: Waiting for Release [ 12075 ]	New: Closed [ 6 ]

Esteban Casuscelli made changes - 05/Jul/2022 9:49 AM

Remote Link

New: This issue links to "KRAK-4604 (Bulldog)" [ 659420 ]

Esteban Casuscelli made changes - 05/Jul/2022 9:45 AM

Fix Version/s

New: 5.0.1 [ 101711 ]

Esteban Casuscelli made changes - 05/Jul/2022 9:44 AM

Fix Version/s

New: 4.4.2 [ 101710 ]

Esteban Casuscelli made changes - 05/Jul/2022 9:44 AM

Fix Version/s

New: 4.3.8 [ 101709 ]

Assignee:: Esteban Casuscelli

Reporter:: Esteban Casuscelli

Affected customers:: 0 This affects my team

Watchers:: 5 Start watching this issue

Created:: 21/May/2022 7:52 PM

Updated:: 25/Oct/2023 1:22 PM

Resolved:: 05/Jul/2022 4:06 PM

Details

Description

Issue Summary

Steps to Reproduce

Expected Results

Actual Results

Dominator Tree - Top-4 Dominators

Ceasium Thread

MembershipHolder

Workaround

Attachments

Issue Links

Forms

Activity

People

Dates