Summary

Lucene Search with unlimited PageFilter causes unnecessarily memory allocation for enterprise instances (1M+ issues).

In many places unlimited PageFilter is used in JQL queries (Agile board, plugins such as JQL Tricks). This leads to the allocation of an array which has number of documents items, where number of documents is typically greater than number of issues.

For 1M+ issues the object size usually is greater than G1HeapRegionSize/2 so it's humongous object. Take a look at related issue https://jira.atlassian.com/browse/JRA-63315 to see why humongous objects are painful.

Environment

• instance with 1M+ issues
• JIRA 6.4.13+

Steps to Reproduce

1. Configure JVM to use at most 32g (exclusive, for xms and xmx). It might be easier to use less memory and then the problem should be visible with less issues.
2. Enable GC logging: -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintReferenceGC -XX:+PrintAdaptiveSizePolicy
3. Any search with unlimited PageFilter is fine. It can be Agile board (default filter seems to use unlimited result set) or eg. linkedIssuesInQuery function from JQL Tricks plugin.
4. Single request is not enough. Try to do thousands of them within a minute.
5. Observe in GC logs:

   2016-12-23T09:35:27.143-0500: 84706.348: [GC pause (G1 Humongous Allocation) (young) (initial-mark)
   Desired survivor size 75497472 bytes, new threshold 1 (max 15)
   - age 1: 187032496 bytes, 187032496 total
   - age 2: 41910368 bytes, 228942864 total
   , 0.3112215 secs]

   and eventually frequent Humongous Allocation will cause high memory pressure and could lead to frequent Full GC:

   2016-12-23T09:51:09.273-0500: 85648.478: [Full GC (Allocation Failure) 15G->5439M(20G), 22.7404778 secs]
   [Eden: 0.0B(1120.0M)->0.0B(1336.0M) Survivors: 0.0B->0.0B Heap: 15.8G(22.0G)->5439.5M(20.0G)], [Metaspace: 471107K->471107K(1523712K)]
   [Times: user=35.18 sys=0.02, real=22.74 secs]

   Expected Results

Search does not cause Humongous Allocation without the good reason.

Actual Results

Humongous Allocations are present, and also Full GC (Allocation Failure) may happen slowing down the whole instance.

Notes

As part of the fix, additional logging will be be implemented. To enable it:

• com.atlassian.jira.issue.search.providers.LuceneSearchProvider should be set to DEBUG level.
  It logs only queries which try to fetch more than 1000 documents.

Workaround

For some cases, increasing xms may help to fit into bigger G1HeapRegionSize.