Caching of reports data is causing high memory pressure and performance degradation

Symptoms

With high load of reports/dashboard statistics gadgets and lots of issue updates this issue causes high memory pressure and leads to either degraded performance or in worst case out of memory error.

Root cause

DefaultReaderCache is using IndexReader as a key to cache field values. 

Field values for single field on 2M+ issues instances can occupy more than 250MB of RAM and consist of millions of objects.

DefaultReaderCache is using weak keys and soft values to store data. Weak keys uses identity comparison on objects.

IndexReader is reopened after every issue write operation on JIRA and this causes all segment readers to be cloned and as result the values from DefaultReaderCache cannot be used any more. This is happening for all segments even those that haven't been changed.  

Fix

As intermediate fix we will use reader.getCoreCacheKey() as key for the cache. More complex approach with limiting number of stored objects will be prepared in future releases.