Extensive memory allocation for indexes with large number of terms

Summary

Indexes with large number of terms (unique pairs of field name and it's values) cause extensive memory allocation each time an issue is being reindexed. On every commit to Lucene index TermInfosReader is regenerated. It contains several large arrays: terms, term infos and index pointers. This situation leads to frequent humongous allocations on G1-enabled instances.

Steps to reproduce

Provide instance with index containing large amount of unique terms (millions) and reindex issues several times.

Actual results

Each commit to Lucene index causes extensive memory allocation.
On G1-enabled instances it can lead to humongous allocations.

Expected results

Commit to Lucene index does not lead to humongous allocations.

Notes

Allocation rates observed on large instances:

• Number of terms in Issues index: more than 100mln
• Lucene builds terms index for every 128th term
• Number of objects in each of TermInfosReader's arrays: 100mln / 128 = ~780k
• Retained heap observed in heap dumps: ~130mb

Workaround

There is no workaround for this issue.