-
Type:
Bug
-
Resolution: Fixed
-
Priority:
Low
-
Affects Version/s: 9.11.0, 9.12.2, 9.12.12
-
Component/s: App Diagnositcs
-
9.11
-
3
-
Severity 2 - Major
-
17
Issue Summary
After upgrading Jira to 9.11 or higher, Jira's in-product diagnostics has the user.directory.connection.latency performance metric.
This metric works by calling the configured LDAP user query for the directory and checking the time in which it returns.
Even though it doesn't get all of the response pages, depending on the resources allocated to the LDAP server (or Active Directory Domain Controller) one query per node per minute could potentially cause high CPU usage.
This is reproducible on Data Center: yes
Steps to Reproduce
- Create a large LDAP directory and allocate few CPUs to its server
- Set up Jira with several nodes (5, for example)
- Have Jira synchronize with the LDAP directory
Expected Results
After the full sync ends, no high CPU is observed in the LDAP directory.
Actual Results
The LDAP directory has high CPU constantly.
In Jira, we often see the IPD threads (ipd-worker:thread-#) in TIMED_WAITING with a stack like this:
jdk.internal.misc.Unsafe.park(java.base@11.0.21/Native Method) java.util.concurrent.locks.LockSupport.parkNanos(java.base@11.0.21/LockSupport.java:234) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(java.base@11.0.21/AbstractQueuedSynchronizer.java:2123) java.util.concurrent.LinkedBlockingQueue.poll(java.base@11.0.21/LinkedBlockingQueue.java:458) com.sun.jndi.ldap.LdapRequest.getReplyBer(java.naming@11.0.21/LdapRequest.java:120) com.sun.jndi.ldap.Connection.readReply(java.naming@11.0.21/Connection.java:443) com.sun.jndi.ldap.LdapClient.getSearchReply(java.naming@11.0.21/LdapClient.java:639) com.sun.jndi.ldap.LdapClient.search(java.naming@11.0.21/LdapClient.java:562) com.sun.jndi.ldap.LdapCtx.doSearch(java.naming@11.0.21/LdapCtx.java:2014) com.sun.jndi.ldap.LdapCtx.searchAux(java.naming@11.0.21/LdapCtx.java:1873) com.sun.jndi.ldap.LdapCtx.c_search(java.naming@11.0.21/LdapCtx.java:1798) com.sun.jndi.toolkit.ctx.ComponentDirContext.p_search(java.naming@11.0.21/ComponentDirContext.java:392) com.sun.jndi.toolkit.ctx.PartialCompositeDirContext.search(java.naming@11.0.21/PartialCompositeDirContext.java:358) javax.naming.directory.InitialDirContext.search(java.naming@11.0.21/InitialDirContext.java:276) ... com.atlassian.crowd.directory.ldap.SpringLdapTemplateWrapper$3.timedGet(SpringLdapTemplateWrapper.java:139) com.atlassian.crowd.directory.ldap.monitoring.TimedSupplier.get(TimedSupplier.java:37) com.atlassian.crowd.directory.ldap.SpringLdapTemplateWrapper.invokeWithContextClassLoader(SpringLdapTemplateWrapper.java:85) com.atlassian.crowd.directory.ldap.SpringLdapTemplateWrapper.search(SpringLdapTemplateWrapper.java:139) com.atlassian.crowd.directory.SpringLDAPConnector.pageSearchResults(SpringLDAPConnector.java:340) com.atlassian.crowd.directory.SpringLDAPConnector.searchEntitiesWithRequestControls(SpringLDAPConnector.java:399) com.atlassian.crowd.directory.SpringLDAPConnector.searchEntities(SpringLDAPConnector.java:383) com.atlassian.crowd.directory.SpringLDAPConnector.searchUserObjects(SpringLDAPConnector.java:586) com.atlassian.crowd.directory.SpringLDAPConnector.searchUsers(SpringLDAPConnector.java:933) com.atlassian.jira.ipd.userdirectory.service.UserDirectoryConnectionServiceImpl.getLatency(UserDirectoryConnectionServiceImpl.java:68) ... com.atlassian.jira.ipd.userdirectory.ExtUserDirectoryLatencyIpdJob.generateMetrics(ExtUserDirectoryLatencyIpdJob.java:52) com.atlassian.jira.ipd.userdirectory.ExtUserDirectoryLatencyIpdJob.runJob(ExtUserDirectoryLatencyIpdJob.java:47)
If we set com.atlassian.crowd.directory to DEBUG, we can see something like this every minute in the logs:
2024-02-06 19:59:44,878+0000 ipd-worker:thread-1 DEBUG [c.a.crowd.directory.SpringLDAPConnector] Performing user search: baseDN = DC=atl-cd,DC=net - filter = (&(&(objectCategory=Person)(sAMAccountName=*))(|(accountExpires=0)(!(accountExpires=*))(accountExpires>=133517231848780000))) in directory 10000 2024-02-06 19:59:44,879+0000 ipd-worker:thread-1 DEBUG [c.a.crowd.directory.SpringLDAPConnector] Paged results are enabled with a paging size of: 1000 2024-02-06 19:59:44,879+0000 ipd-worker:thread-1 DEBUG [c.a.c.d.ldap.monitoring.TimedSupplier] Execute operation search with handler on baseDN: DC=atl-cd,DC=net, filter: (&(&(objectCategory=Person)(sAMAccountName=*))(|(accountExpires=0)(!(accountExpires=*))(accountExpires>=133517231848780000))) 2024-02-06 19:59:44,913+0000 ipd-worker:thread-1 DEBUG [c.a.c.d.ldap.monitoring.ExecutionInfoNameClassPairCallbackHandler] The operation returned 33 results 2024-02-06 19:59:44,914+0000 ipd-worker:thread-1 DEBUG [c.a.c.d.ldap.monitoring.TimedSupplier] Timed call for search with handler on baseDN: DC=atl-cd,DC=net, filter: (&(&(objectCategory=Person)(sAMAccountName=*))(|(accountExpires=0)(!(accountExpires=*))(accountExpires>=133517231848780000))) took 34ms 2024-02-06 19:59:44,914+0000 ipd-worker:thread-1 DEBUG [c.a.crowd.directory.SpringLDAPConnector] Results fetched so far: 33 2024-02-06 19:59:44,914+0000 ipd-worker:thread-1 DEBUG [c.a.crowd.directory.SpringLDAPConnector] Has more results: false 2024-02-06 19:59:44,914+0000 ipd-worker:thread-1 DEBUG [c.a.crowd.directory.SpringLDAPConnector] Cookie length: null
Workaround
Disable in-product diagnostics:
- From the top navigation bar select Administration
> System. - In the left-side panel, go to System Support and select Monitoring.
- Use the In-product diagnostics monitoring toggle to disable IPD monitoring.