Improve FieldCache memory utilization for Jira instances with large Lucene

Problem Definition

Lucene uses org.apache.lucene.search.FieldCache to improve sort for text fields.
Lucene loads corresponding structures from disk into memory and stores in FieldCache.

That means the bigger the Lucene Fields (including Jira CustomFields) size the more memory will be required for them in JVM heap

• For large Jira instances (4+ mln issues) FieldCache might use up to 9GB+, which adds extra pressure for the JVM

Suggested Solution

Optimize memory usage of the FieldCache.
According to Lucene docs, those structures are gone in latest versions, so upgrade of Lucene 7+ should fix the problem.

Jira 8.0 no longer uses field caches to store lucene index parts in Java heap. They migrated this functionality to sth called Doc Values which does exactly the same but does not consume JVM heap space - it's all stored on disk and mmaped into process virtual memory.

Note

Related bog
https://community.developer.atlassian.com/t/where-is-fieldcache-in-jira-8-0/28042

Workaround

Since memory usage is proportional to the size of the Lucene Fields, there are a couple of workarounds here, in the order of preference:

1. Disable Lucene search index for unused CustomFields
   • You can use the queries provided in this KB to identify the unused custom fields.
   • In order to disable the Lucene search index for these fields, you can 'Edit' the custom filed and change the 'Search Template' to 'None', the Lucene search index for this custom field will be disabled.
2. Reduce scope for CustomFields (limit for specific projects)
   • You can find the custom fields with global contexts and then change them to specific projects by following the steps mentioned in this KB. Try applying the changes in batches and during off-peak hours, since when one of your users opens an issue that uses a certain custom field, the cache will need to be recreated.