-
Bug
-
Resolution: Fixed
-
Medium (View bug fix roadmap)
-
None
-
None
Summary
Every load of the Rapid Boards retrieves the sprint custom field ID from the database. The way the code is structured is such that the method used to retrieve the data is synchronised - this means any latency on that retrieval can cause significant thread congestion, presenting in an instance appearing to be unresponsive or experiencing performance problems.
Steps to Reproduce
Load the rapid view board under high load.
Expected Results
The board consistently loads without error.
Actual Results
One of the database retrieval operations may take some time to complete, if this occurs congestion can apply to a number of threads. In worse-case 10 to 100s of threads may block waiting for the operation to complete.
Verification
Review the thread dumps to verify if there is congestion on the com.atlassian.greenhopper.service.sprint.SprintCustomFieldServiceImpl monitor, specifically there are a number of threads waiting for it as per the attached screenshot. They can also be analysed with the following:
grep "waiting to lock.*SprintCustomFieldServiceImpl" jira_threads* | awk '{print $1}' |sort |uniq -c
1 jira_threads.1424257020.txt:
42 jira_threads.1424689917.txt:
In jira_threads.1424689917.txt we can see 42 threads are waiting - this would imply that thread dump is affected with that problem.
Notes
Upgrading to JIRA 6.2 or higher has been known to improve the problem.
- is blocked by
-
JSWSERVER-10819 Cache frequently used PropertySets
- Gathering Interest
- mentioned in
-
![[Extranet] Page](/images/icons/generic_link_16.png "[Extranet] Page")
Page
Failed to load
-
Page
Failed to load
-
Page
Failed to load
-
Page
Failed to load
-
Page Loading...
-
- was cloned as
-
[JSWSERVER-10825] Refactor SprintCustomFieldManagerImpl to use more fine-grained locking
With JIRA 6.2, would this show as more than usual DB connections being held open? And would increasing the total number of Tomcat threads help?
David Marshall
added a comment - We're seeing this with JIRA Server 6.2.7 and JIRA Agile 6.6.51.
$ grep "waiting to lock.*SprintCustomFieldServiceImpl" jstack.* | awk '{print $1}' | sort | uniq -c
18 jstack.0:
18 jstack.1:
27 jstack.2:
47 jstack.3:
87 jstack.4:
95 jstack.5:
120 jstack.6:
160 jstack.7:
164 jstack.8:
173 jstack.9:
David Marshall
added a comment - We're seeing this with JIRA Server 6.2.7 and JIRA Agile 6.6.51.
$ grep "waiting to lock.*SprintCustomFieldServiceImpl" jstack.* | awk '{print $1}' | sort | uniq -c
18 jstack.0:
18 jstack.1:
27 jstack.2:
47 jstack.3:
87 jstack.4:
95 jstack.5:
120 jstack.6:
160 jstack.7:
164 jstack.8:
173 jstack.9:
We see the number of threads as reaching maxThreads just about when we grind to a halt, current maxThreads is 500. Our pool-min-size is 100 and max-pool-size is 500, our maximum number of connections has been 137. In addition to all of the threads locked for SprintCustomFieldManagerImpl, our Java expert noticed that there are many threads are locked on org.apache.catalina.loader.WebappClassLoader.loadClass, and he said "Ask them if they can load their plugins before someone asks instead of on demand while holding a lock."