Issue Summary

During Jira Data Center upgrades to 10.3 or higher, Jira may fail to start or experience extreme startup delays due to a Java heap space OutOfMemoryError. The problem occurs in an Active Objects (AO) upgrade task within the Atlassian Troubleshooting and Support Tools (ATST) plugin: AODataCacheSchemaFixUpgradeTask.

This task attempts to load the entire AO_4789DD_DCDP_HI_CACHE table into memory. On large enterprise instances where this diagnostics cache contains millions of rows, the AO upgrade exhausts the JVM heap and causes a “GC death spiral” or java.lang.OutOfMemoryError: Java heap space in the active-objects-init / I/O dispatcher threads. As a result, the upgrade either never completes or takes an unacceptably long time, blocking production upgrades.

The only viable remediation today is a DBA‑level manual truncation of the cache table before/during the upgrade, which is documented in the internal KB above.

Steps to Reproduce

• Start from a Jira Data Center 10.3.16 instance with ATST enabled.

• Allow the ATST health indicator cache to grow until AO_4789DD_DCDP_HI_CACHE contains several million rows (large, long‑running production instance).

• Upgrade Jira to a later 10.x version where the AO task AODataCacheSchemaFixUpgradeTask is introduced.

• Start Jira and wait for the application to run plugin/AO upgrade tasks.

• Depending on Heap size and table size, you can bbserve Jira entering prolonged startup / “GC death spiral” and failing with java.lang.OutOfMemoryError while the AO upgrade is operating on AO_4789DD_DCDP_HI_CACHE.

Expected Results

• Jira upgrade should complete successfully without exhausting heap memory, regardless of the size of the ATST diagnostics cache table.

• The AO upgrade task should process AO_4789DD_DCDP_HI_CACHE in a memory‑safe manner (e.g., paginated / streaming reads, or safe cleanup/compaction of old data) and not require manual database intervention.

Actual Results

During startup on the upgraded version, Jira:

• Spends a long time in AO initialization / plugin upgrade.
• Eventually hits java.lang.OutOfMemoryError: Java heap space in active-objects-init (or related I/O dispatcher threads).
• May never complete startup without manual DBA action.
• Within the heapdump if captured, we see this. The dominator tree shows that objects retained by java.lang.Thread account for the majority of the heap. From the stack traces, we can see that the active-objects-init-0 thread is running the following AO upgrade task during startup:

  • com.atlassian.troubleshooting.dcdp.transfer.data.cache.ao.AODataCacheSchemaFixUpgradeTask.upgrade(...) 

Workaround

• Stop Jira on all nodes.
• Take a full database backup.
• (Optional) Backup the table to a non‑AO name:

CREATE TABLE BACKUP_DCDP_HI_CACHE AS SELECT * FROM AO_4789DD_DCDP_HI_CACHE;

• Truncate the ATST cache table:

TRUNCATE TABLE AO_4789DD_DCDP_HI_CACHE;

• Start Jira again and confirm that the AO upgrade task completes and Jira starts successfully.
• The table will be repopulated automatically with fresh transient diagnostics data after startup.