Issue Summary

Issue JSON generation is decelerated when custom field contexts are excessively scoped to specific projects. This component is used both in the REST API and issue-related webhooks for synchronous and asynchronous payload generation.

During generation, the JSON serializer needs to determine which navigable fields the requesting user can view, based on custom field contexts and project permissions. This process iterates through all unhidden navigable fields, evaluating the contexts and each project specified within the context scheme.

Stack trace

Single-node DC

com.atlassian.cache.memory.DelegatingCachedReference.get(DelegatingCachedReference.java:76)
com.atlassian.cache.impl.metrics.InstrumentedCachedReference.get(InstrumentedCachedReference.java:58)
com.atlassian.jira.cache.stats.CachedReferenceWithStats.get(CachedReferenceWithStats.java:24)
com.atlassian.jira.project.CachingProjectManager.getProjectObj(CachingProjectManager.java:80)
com.atlassian.jira.issue.context.ProjectContext.getProjectObject(ProjectContext.java:74)
com.atlassian.jira.issue.fields.config.FieldConfigSchemeImpl.isAllProjects(FieldConfigSchemeImpl.java:177)
[...]
com.atlassian.jira.issue.fields.ImmutableCustomField.isAllProjects(ImmutableCustomField.java:1303)
com.atlassian.jira.issue.fields.DefaultFieldManager.userHasPermissionToCustomFieldProjects(DefaultFieldManager.java:282)
com.atlassian.jira.issue.fields.DefaultFieldManager.getAvailableNavigableFields(DefaultFieldManager.java:264)
[...]
com.atlassian.jira.rest.v2.issue.IssueBeanBuilder2.createFieldsDataFromNavigable(IssueBeanBuilder2.java:277)
com.atlassian.jira.rest.v2.issue.IssueBeanBuilder2.createFieldsData(IssueBeanBuilder2.java:208)
com.atlassian.jira.rest.v2.issue.IssueBeanBuilder2.build(IssueBeanBuilder2.java:145)
com.atlassian.jira.rest.v2.issue.IssueBeanBuilder.build(IssueBeanBuilder.java:63)
com.atlassian.jira.plugins.webhooks.serializer.issue.IssueBeanFactory.createBean(IssueBeanFactory.java:37)

Clustered

com.atlassian.cache.ehcache.DelegatingCachedReference.get(DelegatingCachedReference.java:71)
com.atlassian.cache.impl.metrics.InstrumentedCachedReference.get(InstrumentedCachedReference.java:58)
com.atlassian.jira.cache.stats.CachedReferenceWithStats.get(CachedReferenceWithStats.java:24)
com.atlassian.jira.project.CachingProjectManager.getProjectObj(CachingProjectManager.java:80)
[...]

Steps to Reproduce

An XML site backup and reproduction script has been attached to this bug report. To use it:

1. Install Jira 9.12.24 (backup version) and restore the XML backup. After restoring, upgrade to a later version, if needed.
2. Configure 5+ webhooks (“issue updated” event, no JQL filter).
3. Run the reproduction script (generates 5000 issue updates with 10 parallel threads). Adjust the URL/PAT and call with "node update-issues-parallel-batch.js".
4. Capture thread dumps during execution.

Backup Contents

• 500 projects
• 5000 issues (10/project)
• 500 custom fields, each explicitly including all 500 projects in the context scheme

Expected Results

The HTTP or webhook publisher/executor threads (webhook-dispatcher) can efficiently serialize the issue.

Actual Results

High CPU usage with processing time growing exponentially:

(project cache lookups) =
  (number of custom fields)
  x (number of contexts per custom field)
  x (number of projects in each context)
  [x (number of applicable webhooks)]

= 1,250,000 = 500 x 1 x 500 x 5

This causes delays in webhook payload generation/delivery and issue REST API response times.

Jira 9.12 (sync webhooks)

Using the test dataset, about 46% of the thread dump frames are accessing the project cache:

HTTP threads processing webhooks

HTTP threads processing webhooks -> accessing project cache

Jira 10.3 (async webhooks)

Using the test dataset, about 37% of the thread dump frames are accessing the project cache:

In some customer environments, 95-100% of frames are accessing this cache.

Workaround

Workaround 1 (short term)

Hide unnecessary fields from the field configuration (Specifying field behavior). This excludes the field before it reaches this expensive path.

Note: excluding the current issue's project from the custom field's context (Configuring custom field contexts) doesn't alleviate the problem since the expensive path will still process all other projects in the context.

Workaround 2 (long term)

Decrease the overall number of custom fields. The product guardrail is 1,200 custom fields, and support recommends staying under 700 custom fields.

• Managing custom fields in Jira effectively
• Jira Software guardrails
• Analyzing the usage of custom fields

Workaround 3 (webhooks only)

Increase the webhook executor thread pool size (webhooks.executor.thread.count), as described in Tuning webhook performance parameters in Jira Data Center). This may result in higher CPU utilization.

Similarly, consider reducing the webhook executor queue size (webhooks.executor.queue.size) and increasing the enqueue timeout (webhooks.executor.queue.wait.timeout.millis) to 10-30 seconds. This replicates some of the natural backpressure from synchronous webhooks, which may force users and integrations to wait longer for their edits/transitions to return.

Note: These parameters are only available in Jira 10.3.5+ LTS and 10.6.0+ (not 10.4.x or 10.5.x). Please upgrade to 10.3.5+ or 10.6.0+.