Atlassian Diagnostics plugin unnecessarily loads DETAILS_JSON CLOBs and can cause OutOfMemory error

XMLWordPrintable

    • Type: Bug
    • Resolution: Unresolved
    • Priority: Medium
    • 11.3.9
    • Affects Version/s: 10.0.0, 11.0.0, 11.3.0, 11.3.8
    • Component/s: App Diagnositcs, JQL Search
    • None
    • 10
    • 2
    • Severity 2 - Major
    • 41

      Issue Summary

      DefaultAlertEntityDao.streamMetrics() aggregates alert metadata but selects DETAILS_JSON through AlertAO.ALL_COLUMNS, even though it never reads that field.

      When alert details are very large, this unnecessary projection can cause excessive heap allocation and OOME during alert analytics.

      This projection issue is longstanding and existed as far as Jira 7. However, JRASERVER-79454 amplifies it in Jira 11.x by creating much larger diagnostics alert details.

      Steps to Reproduce

      1. Create or retain large DETAILS_JSON values in AO_C16815_ALERT_AO, for example through a plugin JQL function returning a very large issue set
      2. Run the daily alert analytics job or invoke streamMetrics()
      3. Observe that the query selects DETAILS_JSON despite metrics aggregation needing only alert metadata
      4. With enough large rows, observe excessive heap allocation and possible OOME

      Expected Results

      streamMetrics() selects only the columns required to aggregate alert metrics and does not select DETAILS_JSON.

      Actual Results

      streamMetrics() selects ALL_COLUMNS, including DETAILS_JSON. Large alert details are loaded unnecessarily during metrics aggregation and can cause OOME.

      Relationship to JRASERVER-79454

      • This issue is the unnecessary DETAILS_JSON projection in alert analytics
      • JRASERVER-79454 makes the stored alert details much larger
      • Together, they can cause OOME
      • Fixing this issue protects instances that already contain oversized rows

      Workaround

      • Truncate the AO_C16815_ALERT_AO table to remove the large entries
        TRUNCATE TABLE "AO_C16815_ALERT_AO"
        
      • Decrease the DB retention period for the diagnostic entries from the 30-days default by adding the JVM parameter:
        -Dcom.atlassian.jira.health.diagnostics.alerts.retention-period-days=7
        
      • Alternatively, increasing jira.diagnostics.thresholds.slow-query-millis reduces how often JQL diagnostics alerts are recorded.
      • As a last resort, the JQL diagnostics monitor can be turned off to stop these alerts from being recorded.
        As a system administrator, disable the Dark Feature com.atlassian.diagnostics.jql.monitor

              Assignee:
              Mateusz Beczek
              Reporter:
              Mateusz Beczek
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Created:
                Updated: