-
Bug
-
Resolution: Fixed
-
Medium
-
2.5.9
-
13
-
Severity 2 - Major
-
11
-
Summary
When using Service Desk 2.5.9 with an Agent based license the application response time is severely impacted when there are more than 10k Agents and during LDAP syncs. The worst affected views are from the REST APIs, rest/api/2/users and rest/api/2/groups.
After upgrade, 99th percentile of all request times increased to 30 seconds when Active Directory user synchronization is running or when our jira-ilists group (17k users) is added to Service Desk Agent role. Otherwise, 99th% is ~2-5 sec. Monitoring 99th percentile execution time of all requests has had strong correlation with user complaints about performance in the past.
We have unlimited licenses for everything. When we upgraded to JIRA 6.4 and JIRA Service Desk 2.5.9 we were issued an Agent-base license (unlimited Agents). Previously all jira-users could access SD features we wanted without extensive intervention from JIRA admins. Our users mainly want the simplified issue creation and SLAs.
With an Agent based license, only Agents that are members of the Service Desk Team role for a project can be assigned issues and this is a big obstacle for us. To work around it we added the equivalent of our jira-users group to the Service Desk Agent group (jira-ilists, approx 17k users). This triggered the performance problems.
Environment
- JIRA 6.4.12, Service Desk 2.5.9
Steps to Reproduce
- Install JIRA 6.4.12 with Service Desk 2.5.9
- Generate 400 projects, each with a Service Desk.
- Add a group with 10k members to the Service Desk Agent group.
Expected Results
- Requests for rest/api/2/users and rest/api/2/groups should perform well during Active Directory sync and when using an Agent based license.
Actual Results
- Requests for rest/api/2/users and rest/api/2/groups are dramatically slower when there are thousands of Agents with an Agent based license.
Notes
Extensive logs and data are available on the support case, PS-3211. This is only accessible by Atlassians.
Workaround
- Only add users that require Agent features to the Service Desk Agents group rather than all users.
- is related to
-
-
- Gathering Impact
-
- relates to
-
-
- Closed
-
- mentioned in
-
Page Failed to load
-
Page Failed to load
-
Page Failed to load
-
Page Loading...
-
-
-
- was cloned as
-
| Form Name | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
[JSDSERVER-3357] JIRA performance is degraded when using an Agent based license with lots of Agents
David Marshall
added a comment - I suspect that one of the reasons that rest/api/2/users and rest/api/2/groups performed poorly was because there was simultaneous Service Desk activity in which agent access was being checked. We experienced even worse performance degradation when we were synchronizing our Active Directory user directory.
It might be relevant to the issue that we had other groups than service-desk-agents in the global JIRA Service Desk agent access permission.
One thing that piqued my interest was these rows in cwd_group_attributes. Both groups (11010 and 11011) are named service-desk-agents.
*************************** 1. row ***************************
ID: 10000
group_id: 11010
directory_id: 1
attribute_name: synch.created.by.jira.service.desk
attribute_value: synch.created.by.jira.service.desk
lower_attribute_value: synch.created.by.jira.service.desk
*************************** 2. row ***************************
ID: 10001
group_id: 11011
directory_id: 10000
attribute_name: synch.created.by.jira.service.desk
attribute_value: synch.created.by.jira.service.desk
lower_attribute_value: synch.created.by.jira.service.desk
Those rows suggest the hypothesis that the group service-desk-agents is tightly coupled to the user directories. If the user directories are being updated at the same time that there's a considerable amount of Service Desk activity, then resource contention affects performance rather significantly.
David Marshall
added a comment - I suspect that one of the reasons that rest/api/2/users and rest/api/2/groups performed poorly was because there was simultaneous Service Desk activity in which agent access was being checked. We experienced even worse performance degradation when we were synchronizing our Active Directory user directory.
It might be relevant to the issue that we had other groups than service-desk-agents in the global JIRA Service Desk agent access permission.
One thing that piqued my interest was these rows in cwd_group_attributes . Both groups (11010 and 11011) are named service-desk-agents .
*************************** 1. row ***************************
ID: 10000
group_id: 11010
directory_id: 1
attribute_name: synch.created.by.jira.service.desk
attribute_value: synch.created.by.jira.service.desk
lower_attribute_value: synch.created.by.jira.service.desk
*************************** 2. row ***************************
ID: 10001
group_id: 11011
directory_id: 10000
attribute_name: synch.created.by.jira.service.desk
attribute_value: synch.created.by.jira.service.desk
lower_attribute_value: synch.created.by.jira.service.desk
Those rows suggest the hypothesis that the group service-desk-agents is tightly coupled to the user directories. If the user directories are being updated at the same time that there's a considerable amount of Service Desk activity, then resource contention affects performance rather significantly.
I see no evidence of such behaviour on latest Jira versions. Please reach out to support if you are still experiencing similar issues, and we will be happy to assist.