Uploaded image for project: 'Jira Server and Data Center'
  1. Jira Server and Data Center
  2. JRASERVER-67805

Improve SegmentNorms memory utilisation for Jira instances with large Lucene

    XMLWordPrintable

    Details

    • UIS:
      20
    • Support reference count:
      4
    • Feedback Policy:
      We collect Jira feedback from various sources, and we evaluate what we've collected when planning our product roadmap. To understand how this piece of feedback will be reviewed, see our Implementation of New Features Policy.

      Description

      Problem Definition

      Lucene uses org.apache.lucene.index.SegmentNorms for Fulltext search.
      Initially, those data structures are created as files on disk:

      ls -al caches/indexes/issues/*.nrm
      181030 caches/indexes/issues/_b4w.nrm
      

      and then later Lucene loads them into memory and stores in SegmentNorms.

      That means the bigger the Lucene Norm size the more memory will be required for them in JVM heap

      • For large Jira instances (4+ mln issues) SegmentNorms might use up to 6GB+, which adds extra pressure for the JVM
      • They're recalculated on each write to the index, due to JRASERVER-67125, so it's highly advisable to update to the corresponding version.

      Heap dump details

      Example of histogram:

      Class Name Objects Shallow Heap Retained Heap
      org.apache.lucene.index.ReadOnlySegmentReader 167 14.696 >= 3.998.311.056
      org.apache.lucene.index.SegmentNorms 308.254 17.262.224 >= 3.828.471.112

      Dominator tree:

      searcher: org.apache.lucene.search.IndexSearcher
      reader: org.apache.lucene.index.ReadOnlyDirectoryReader which loads org.apache.lucene.index.ReadOnlySegmentReader
      which uses org.apache.lucene.index.SegmentNorms

      Suggested Solution

      Optimize memory usage of the SegmentNorms.
      According to Lucene docs, those structures are gone in latest versions, so upgrade of Lucene should fix the problem.

      Workaround

      Since memory usage is proportional to number of Lucene docuements and size of the Norms, there are a couple of workarounds here, in the order of preference:

      1. Disable Lucene search index for unused CustomFields
      2. Reduce scope for CustomFields (limit for specific projects)
      3. Keep Norms smalls by running full reindex periodically

        Attachments

          Issue Links

            Activity

              People

              Assignee:
              Unassigned
              Reporter:
              ayakovlev@atlassian.com Andriy Yakovlev
              Votes:
              5 Vote for this issue
              Watchers:
              11 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved: