Details
-
Type:
Bug
-
Status: Gathering Impact (View Workflow)
-
Priority:
Low
-
Resolution: Unresolved
-
Affects Version/s: 7.13.13
-
Fix Version/s: None
-
Component/s: Issue - Create Issue
-
Labels:None
-
Introduced in Version:7.13
-
Support reference count:2
-
Symptom Severity:Severity 1 - Critical
-
UIS:3
-
Bug Fix Policy:
Description
Issue Summary
We had an outage on a Jira instance running JSD 3.16.11 today which appeared to be caused by requests to the `/jira/rest/internal/2/user/mention/search` endpoint.
Looking at request times for different queries, those that do NOT contain a query string appear to be significantly slower.
Requests WITHOUT a query string:
Note the scale change - the longest with a query string was 13s, the shortest without was 28s, with longest over 60s
Examples:
- With a query string, took 0.14s: /jira/rest/internal/2/user/mention/search?issueKey=KEY-5678&maxResults=10&query=il&_=1587494420052, (referrer is /jira/browse/KEY-5678)
- Without a query string, took 62s: /jira/rest/internal/2/user/mention/search?issueKey=KEY-1234&maxResults=10&_=1587407226218 (referrer is /jira/browse/KEY-1234?filter=-1)
Steps to Reproduce
I am not sure what this endpoint is - I went into the referrer (for a lot of them this was filter=-1) and clicked around but was not able to hit that endpoint with anything I did.
Expected Results
Searches return promptly with no performance impact.
Actual Results
This actually caused an outage for us due to a spike in CPU which seems connected to GC activity.
New Relic showing that this endpoint is the slowest:
CPU activity by thread showing that GC is the main consumer of CPU. NB: we got the threads long after the request returned/timed out - it is likely still processing in the back end at this point:
Workaround
Currently there is no known workaround for this behavior. A workaround will be added here when available
Attachments
Issue Links
- is related to
-
JSWSERVER-20336 Searching and Mentioning users may cause performance issues and high CPU load
-
- Gathering Impact
-
-
JSDSERVER-6808 Slowly typing query into JSD Project Workload search causes service unavailability
-
- Gathering Impact
-
-
JSDSERVER-6047 Workload report loads very slowly
-
- Short Term Backlog
-