Label suggestions performance grows slow linearly as Jira scales

Issue Summary

When the number of labels is high, it takes 5+ seconds to load suggestions.
It is checked on GSAC and it takes around 20 seconds to load label suggestions
This is reproducible on Data Center: Yes

Steps to Reproduce

1. Create 200K labels for different issues
2. Open an issue
3. Click the Pencil icon to edit Labels
4. Click the arrow to see label suggestions
   More details around the reproduction steps in Internal comment

The same happens if you configure an Advanced Roadmaps Plan and add the Labels field to the Plan. Upon clicking on the Labels cell, Jira will fire a request to /rest/jpo/1.0/labels that'll take too long to respond.

Expected Results

The Label suggestions should be listed immediately.

Actual Results

• From thread dumps, it looks like we’re we’re loading ALL the UNIQUE labels first - followed by sorting the most used ones:

          at com.atlassian.jira.web.bean.StatisticAccessorBean.getWrapper(StatisticAccessorBean.java:126)
  	at com.atlassian.jira.web.bean.StatisticAccessorBean.getAllFilterBy(StatisticAccessorBean.java:283)
  	at com.atlassian.jira.issue.label.suggestions.PopularLabelsProvider.findMostPopular(PopularLabelsProvider.java:20)
  	at com.atlassian.jira.issue.label.suggestions.LabelSuggester.getSuggestions(LabelSuggester.java:55)
  	at com.atlassian.jira.issue.label.suggestions.LabelSuggester.getSuggestedLabels(LabelSuggester.java:26)
  	at com.atlassian.jira.issue.label.DefaultLabelManager.getSuggestedLabels(DefaultLabelManager.java:149)
  	at com.atlassian.jira.bc.issue.label.DefaultLabelService.getSuggestedLabels(DefaultLabelService.java:178)
  

• As Jira scales (No. of labels and No. of issues the labels are used in) the response time to load the suggested labels grows linearly slower:
  

Workaround

The only currently workaround available is to identify the expensive and CPU consuming Label autocomplete suggestions requests to Jira and block them either on the Proxy / Load Balancer or in Jira's urlrewrite:

• How to block Label type fields autocomplete suggestions and other requests in Jira

Those are usually the requests with empty values (0 inputted characters).

Note:

If you come across this issue - please comment on this bug with the following SQL outputs which could further help us with the size profiles general customer environments:

• SELECT COUNT(*) FROM label;

• SELECT COUNT(DISTINCT l.label) FROM label l;

• SELECT count(*) from "label" l group by l."label" order by count DESC limit 10;