-
Bug
-
Resolution: Fixed
-
Medium (View bug fix roadmap)
-
3.13.5, 4.1.1, 5.0, 5.2.5, 6.0.8
-
None
-
3.13
-
Summary
When a session contains a search result (as a result of using any search in JIRA), there is a possibility that during standard view issue operations a bottleneck can occur on the NextPreviousPager monitor. This can result in the entire JIRA becoming unresponsive and rendering the instance unusable.
Steps to Reproduce
It is not currently known what the exact steps are to reproduce this, however it involves viewing issues when a search result is contained within a user session.
Expected Results
JIRA operates as normal.
Actual Results
JIRA becomes unresponsive as lock congestion occurs on the NextPreviousPager monitor.
Notes
View issue screen is pretty fast (my measures points to 16 RPS), but only we have no active search. If the session contain a search request, updating NextPreviousPager is a significant bottleneck.
Even on 1 RPS response time raise exponentially and after 1 minute of load JIRA became unresponsive.
The performance impact can be exacerbated by using complicated searches or expensive JQL functions.
Diagnosis
Generate a Thread Dump 6 times every 10 seconds during periods of unresponsiveness and read it with TDA or a similar tool. If there is a bottleneck in the NextPreviousPager monitor as per the attached screenshot there is a significant likelihood the instance is affected by this problem.
This can also be identified with the below commands - we're going to grep the thread dumps to see how many threads that are waiting to lock NextPreviousPager. If there's a high number persisting across multiple thread dumps over time, it's probably this bug.
$ grep "waiting to lock.*NextPreviousPager" jstack.4440.threadHigh.* | awk '{print $1}' |sort |uniq -c
24 jstack.4440.threadHigh.103649.939600529:
27 jstack.4440.threadHigh.103701.620551473:
25 jstack.4440.threadHigh.103713.178859435:
25 jstack.4440.threadHigh.103724.736087099:
25 jstack.4440.threadHigh.103736.376659166:
This shows the number of threads waiting to lock the monitor grouped by thread dump. In each thread dump that's 25+ which is a large number. Or read the threadies and check for larger numbers of this:
"http-bio-8080-exec-1479" daemon prio=10 tid=0x00007f3fe07ba800 nid=0x6d4f waiting for monitor entry [0x00007f3e7c581000] java.lang.Thread.State: BLOCKED (on object monitor) at com.atlassian.jira.issue.pager.NextPreviousPager.update(NextPreviousPager.java:151) - waiting to lock <0x00007f46331c6bc8> (a com.atlassian.jira.issue.pager.NextPreviousPager) at com.atlassian.jira.issue.pager.PagerManager.updatePager(PagerManager.java:41)
- is a regression of
-
-
- Closed
-
- is duplicated by
-
-
- Closed
-
- causes
-
JDEV-30233 Failed to load
- copied to
-
JRADEV-13120 Failed to load
- relates to
-
JRADEV-22311 Loading...
| Form Name | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
[JRASERVER-21557] NextPreviousPager is bottleneck under highload
Guys, hello. As a reporter i'm interested, why so much attention to issue which was originally ignored and closed as irrelevant?
The fix for this is to remove the NextPreviousPager completely. It is not needed anymore since JIRA 6.0 but there are still some references to it within DefaultIssueMetadataHelper.java and initSetSelectedIssue.js.
This absolutely murders our system.
We've use the JQL tricks plugin which add some useful, but performance-intensive functions to JQL.
Some of the queries using those functions take a few minutes to run.
When we try and click any of the issues returned in the results, it takes the same amount of time to open the issue as it took to run the query!
The query itself only returns 4 results.
This shouldn't be the case.
I should add we discovered this using our Jira 5.0RC2 test system with JQL Tricks 3.0
NextPreviousPager does have some degenerate performance characteristics. In "normal" situations it performs okay – for example when you are viewing issues sequentially using the next/previous links. Then again, as soon as you start viewing other issues, the problem arises.
It will perform badly if you do a search, and then visit random issues (either within the search, or outside of the search). This behaviors would be exagerated in a performance test if, for example, all your requests shared a session.
Andrey,
What version of JIRA did you observe this on?
(I would guess JIRA 3.13.5 from your user display name - but best to be explicit)
Please read this comment: http://jira.atlassian.com/browse/JRA-13296?focusedCommentId=198223&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#action_198223
We have received contacts from other customers affected by the issue. Fortunately it's resolved in the 6.1.9 and later versions but some customers have not yet been able to upgrade.
Cheers,
Morgan Knicely
Atlassian Support - Austin, TX